Releases: drivendataorg/cloudpathlib
Releases · drivendataorg/cloudpathlib
Release list
v0.25.0
- Security fix (GHSA-r4f8-3xc4-c8vw): local path traversal via
..in cloud object keys.
Cloud object keys are opaque strings and some backends (e.g. Google Cloud Storage) accept..
segments. cloudpathlib mapped keys onto local paths with plain path arithmetic and no containment
check, so a key such as../../evilcould escape the local cache directory (on any
read/write/open/os.fspathof aCloudPath) or a download destination
(download_to/copytree/copy/moveto a local path), writing attacker-controlled bytes
outside the intended directory. cloudpathlib now confirms each computed local target stays within
its base directory and raisesCloudPathLocalPathTraversalErrorotherwise. This guards the cache
mapping (CloudPath._localand its S3/HTTP overrides), both branches ofCloudPath.download_to
andCloudPath.copytree(including joins that can escape via\or drive letters in keys on
Windows), and thecloudpathlib.localmock's storage-side mapping. Reported by mohammad adnan
(cystack.ps redteam). - Added a
lazykeyword argument toCloudPath.walk. By default (lazy=False) the existing fast behavior is preserved: the whole subtree is fetched up front with a single recursive listing. Passinglazy=Truelists each directory on demand so that, whentop_down=True, callers can prune subdirectories by modifyingdirnamesin-place (à laos.walk/Path.walk) to skip fetching the contents of those subtrees entirely — dramatically reducing API calls for large, sparsely-traversed trees. (Issue #518, PR #574) - Added presigned URL support to the mock clients in
cloudpathlib.local, soas_url(presign=True)on aLocalS3Path,LocalGSPath, orLocalAzureBlobPathreturns a deterministic URL withexpiresandsignaturequery parameters instead of raisingNotImplementedError. This lets code that generates presigned URLs be exercised in tests that use the mock classes. (PR #572) - Fixed S3 copy and move operations to forward copy-specific extra args such as
CopySourceSSECustomerKey, and addedaddressing_style="virtual"support forS3Client(Issues #500, #527, PR #576) - Changed
S3Client._get_metadatato read object metadata withHeadObjectinstead ofGetObject, sostat,etag, andsizeno longer open the object body. Also fixes aKeyErroronContentLengthagainst S3-compatible gateways that dropContent-LengthfromGetObjectresponses. (Issue #564, PR #565) - Fixed
KeyErroronETagagainst S3-compatible gateways that omitETagfromHeadObjectresponses. (Issue #582, PR #583) - Fixed mypy 2.x type errors in
ClientandCloudPaththat caused CI lint failures (Issue #563, PR #566) - Added
AGENTS.mdwith repository-specific guidance for coding agents covering contributor
workflow, compatibility expectations, test rig and mock usage, live backend validation, and PR
hygiene. (PR #573)
v0.24.0
v0.23.0
- Added support for Python 3.14 (Issue #529, PR #530)
- Changed
CloudPath.copyto have the first parameter namedtargetinstead ofdestinationand added newfollow_symlinksandpreserve_metadatakwargs. Breaking change for users that relied on the first parameter being nameddestinationinstead oftarget. - Added
CloudPath.copy_intoto copy a file or directory into another file or directory. - Added
CloudPath.moveto move a file or directory to another location. - Added
CloudPath.move_intoto move a file or directory into another file or directory. - Added
CloudPathInfoandCloudPath.infoto get information about a file or directory. - Added additional no-op kwargs to
mkdir,touch,glob,rglob,statto match pathlib.
- Changed
v0.22.0
- Fixed issue with GS credentials, using default auth enables a wider set of authentication methods in GS (Issue #390, PR #514, thanks @ljyanesm)
- Added support for http(s) urls with
HttpClient,HttpPath,HttpsClient, andHttpsPath. (Issue #455, PR #468) - Added experimental support for patching the builtins
open,os,os.path, andglobto work withCloudPathobjects. It is off by default; see the new "Compatibility" section in the docs for more information. (Issue #128, PR #322) - Added support for
CloudPath(*parts)to create aCloudPathobject from a list of parts (to matchpathlib.Path). This is a potentially breaking change for users that relied on the second arg being theclientinstead of making it an explicit kwarg. (PR #322)
v0.21.1
v0.21.0
- Removed support for deprecated env var that had a typo (
CLOUPATHLIB_FILE_CACHE_MODE; you should useCLOUDPATHLIB_FILE_CACHE_MODE). - Added support for
timeoutandretrykwargs forGSClient. (Issue #484, PR #485, thanks @Mchristos) - Fixed
CloudPath(...) / otherto correctly attempt to fall back onother's__rtruediv__implementation, in order to support classes that explicitly support the/with aCloudPathinstance. Previously, this would always raise aTypeErrorifotherwere not astrorPurePosixPath. (PR #479) - Added
md5property toGSPath, updated LocalGSPath to includemd5property, updated mock_gs.MockBlob to includemd5_hashproperty. (Issue #482, PR #483) - Fixed an uncaught exception on Azure Gen2 storage accounts with HNS enabled when used with
DefaultAzureCredential. (Issue #486, PR #487) - Removed support for Python 3.8, which was EOL in October 2024. (PR #502)
v0.20.0
- Added support for custom schemes in CloudPath and Client subclases. (Issue #466, PR #467)
- Fixed
ResourceNotFoundErroron Azure gen2 storage accounts with HNS enabled and issue that some Azure credentials do not haveaccount_name. (Issue #470, Issue #476, PR #478) - Added support for Python 3.13 (Issue #472, PR #474):
.full_matchadded.from_uriaddedfollow_symlinkskwarg added tois_fileadded as no-opfollow_symlinkskwarg added tois_diradded as no-opnewlinekwarg added toread_textrecurse_symlinkskwarg added toglobadded as no-oppatternparameter forglobcan be PathLikerecurse_symlinkskwarg added torglobadded as no-oppatternparameter forrglobcan be PathLike.parserproperty added
v0.19.0
- Fixed an error that occurred when loading and dumping
CloudPathobjects using pickle multiple times. (Issue #450, PR #454, thanks to @kujenga) - Fixed typo in
FileCacheModewhere values were being filled by environment variableCLOUPATHLIB_FILE_CACHE_MODEinstead ofCLOUDPATHLIB_FILE_CACHE_MODE. (PR #424, thanks to @mynameisfiber) - Fixed
CloudPathcleanup viaCloudPath.__del__whenClientencounters an exception during initialization and does not create afile_cache_modeattribute. (Issue #372, thanks to @bryanwweber) - Removed support for Python 3.7 and pinned minimal
boto3version to Python 3.8+ versions. (PR #407) - Changed
GSClientto use the nativeexists()method from the Google Cloud Storage SDK. (PR #420, thanks to @bachya) - Changed default clients to be lazily instantiated (Issue #428, PR #432)
- Fixed
download_toto check for the existence of the cloud file (Issue #430, PR #433) - Added env vars
CLOUDPATHLIB_FORCE_OVERWRITE_FROM_CLOUDandCLOUDPATHLIB_FORCE_OVERWRITE_TO_CLOUD. (Issue #393, PR #437) - Fixed
globforcloudpathlib.local.LocalPathand subclass implementations to match behavior of cloud versions for parity in testing. (Issue #415, PR #436) - Changed how
cloudpathlib.local.LocalClientand subclass implementations track the default local storage directory (used to simulate the cloud) used when no local storage directory is explicitly provided. (PR #436, PR #462)- Changed
LocalClientso that client instances using the default storage access the default local storage directory through theget_default_storage_dirrather than having an explicit reference to the path set at instantiation. This means that callingget_default_storage_dirwill reset the local storage for all clients using the default local storage, whether the client has already been instantiated or is instantiated after resetting. This fixes unintuitive behavior wherereset_local_storagedid not reset local storage when using the default client. (Issue #414) - Added a new
local_storage_dirproperty toLocalClient. This will return the current local storage directory used by that client instance.
by reference through the `get_default_ rather than with an explicit.
- Changed
- Refined the return type annotations for
CloudPath.open()to match the behavior ofpathlib.Path.open(). The method now returns specific types (TextIOWrapper,FileIO,BufferedRandom,BufferedWriter,BufferedReader,BinaryIO,IO[Any]) based on the providedmode,buffering, andencodingarguments. (Issue #465, PR #464) - Added Azure Data Lake Storage Gen2 support (Issue #161, PR #450), thanks to @M0dEx for PR #447 and PR #449
v0.18.1
v0.18.0 (yanked)
- Implement sliced downloads in GSClient. (Issue #387, PR #389)
- Implement
as_urlwith presigned parameter for all backends. (Issue #235, PR #236) - Stream to and from Azure Blob Storage. (PR #403)
- Implement
file:URI scheme support forAnyPath. (Issue #401, PR #404)
Note: This version was yanked due to incompatibility with google-cloud-storage <2.7.0 that causes an import error.