Skip to content

WIP: Implement sliced downloads in GSClient - #389

Merged
pjbull merged 3 commits into
drivendataorg:389-live-testsfrom
joconnor-ecaa:feat/gcs-sliced-download
Dec 29, 2023
Merged

pjbull merged 3 commits into
drivendataorg:389-live-testsfrom
joconnor-ecaa:feat/gcs-sliced-download

Conversation

@joconnor-ecaa

@joconnor-ecaa joconnor-ecaa commented Dec 21, 2023 •

Copy link
Copy Markdown
Contributor

Implements sliced object downloads in GSClient using google.cloud.storage.transfer_manager as documented here.

Closes #387


Contributor checklist:

  • I have read and understood CONTRIBUTING.md
  • Confirmed an issue exists for the PR, and the text Closes #issue appears in the PR summary (e.g., Closes #123).
  • Confirmed PR is rebased onto the latest base
  • Confirmed failure before change and success after change
  • Any generic new functionality is replicated across cloud providers if necessary
  • Tested manually against live server backend for at least one provider
  • Added tests for any new functionality
  • Linting passes locally
  • Tests pass locally
  • Updated HISTORY.md with the issue that is addressed and the PR you are submitting. If the top section is not `## UNRELEASED``, then you need to add a new section to the top of the document for your change.

@joconnor-ecaa joconnor-ecaa changed the title feat: Implement sliced downloads in GSClient WIP: Implement sliced downloads in GSClient Dec 21, 2023
@joconnor-ecaa
joconnor-ecaa marked this pull request as ready for review December 29, 2023 09:02
@codecov

codecov Bot commented Dec 29, 2023

Copy link
Copy Markdown

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (41aa53d) 93.8% compared to head (f467b56) 93.5%.
Report is 3 commits behind head on master.

Additional details and impacted files
@@           Coverage Diff            @@
##           master    #389     +/-   ##
========================================
- Coverage    93.8%   93.5%   -0.4%     
========================================
  Files          22      22             
  Lines        1566    1570      +4     
========================================
- Hits         1470    1468      -2     
- Misses         96     102      +6     
Files Coverage Δ
cloudpathlib/gs/gsclient.py 93.1% <100.0%> (-1.4%) ⬇️

... and 2 files with indirect coverage changes

@pjbull
pjbull changed the base branch from master to 389-live-tests December 29, 2023 17:34
@pjbull
pjbull merged commit f65de44 into drivendataorg:389-live-tests Dec 29, 2023
@pjbull

pjbull commented Dec 29, 2023

Copy link
Copy Markdown
Member

Merged into 389-live-tests to open PR #391 that runs the live tests.

pjbull added a commit that referenced this pull request Dec 29, 2023
* WIP: Implement sliced downloads in GSClient (#389)

* feat: Implement sliced downloads in GSClient

* fix: remove unintended import changes

* Mock transfer_manager. Test both worker types.

* Update HISTORY.md

---------

Co-authored-by: Joe O'Connor <60386246+joconnor-ecaa@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

GSPath: Implement sliced object downloads

2 participants