Skip to content

Initial ADLS gen2 support - #453

Merged
pjbull merged 15 commits into
masterfrom
azure-data-lake-gen2
Aug 28, 2024
Merged

pjbull merged 15 commits into
masterfrom
azure-data-lake-gen2

Conversation

@pjbull

@pjbull pjbull commented Jul 17, 2024 •

Copy link
Copy Markdown
Member

Add initial ADLS gen2 support:

  • Accept or create a data_lake_service_client
  • When necessary in AzureBlobClient use the data_lake_service_client API. This does not use the ADLS API everywhere possible since ADLS Gen2 works with the normal blob API in many places. Do this since it is common to enable hierarchical namespaces when creating a blob storage account, not realizing that it needs ADLS APIs for some scenarios.
  • Add the ADLS SDK as a dependency
  • Adds a new Azure test rig for Gen2 if configured
  • Configures ADLS Gen2 testing on Github Actions

This also incorporates the fixes in:

Closes #161


  • Test the different instantiation pipelines
  • Test empty folders on ADLS Gen2
  • Fix mocks to do offline tests for ADLS Gen2 and not
  • Add additionally functionality mentioned in Support Azure Data Lake Storage Gen2 #161

@github-actions

github-actions Bot commented Jul 17, 2024 •

Copy link
Copy Markdown
Contributor

@github-actions
github-actions Bot temporarily deployed to pull request July 17, 2024 19:12 Inactive
@pjbull pjbull changed the title Minimal ADLS gen2 support WIP: Minimal ADLS gen2 support Jul 18, 2024
Comment thread cloudpathlib/azure/azblobclient.py Outdated
@github-actions
github-actions Bot temporarily deployed to pull request July 28, 2024 05:26 Inactive
@codecov

codecov Bot commented Jul 28, 2024 •

Copy link
Copy Markdown

Codecov Report

Attention: Patch coverage is 97.70115% with 2 lines in your changes missing coverage. Please review.

Project coverage is 94.2%. Comparing base (f3605a6) to head (9853b6b).
Report is 1 commits behind head on master.

Files Patch % Lines
cloudpathlib/azure/azblobclient.py 97.4% 2 Missing ⚠️
Additional details and impacted files
@@           Coverage Diff            @@
##           master    #453     +/-   ##
========================================
+ Coverage    93.6%   94.2%   +0.6%     
========================================
  Files          23      23             
  Lines        1665    1728     +63     
========================================
+ Hits         1559    1629     +70     
+ Misses        106      99      -7     
Files Coverage Δ
cloudpathlib/azure/azblobpath.py 93.6% <100.0%> (+0.9%) ⬆️
cloudpathlib/cloudpath.py 94.5% <100.0%> (+0.3%) ⬆️
cloudpathlib/azure/azblobclient.py 96.8% <97.4%> (+2.0%) ⬆️

... and 2 files with indirect coverage changes

@github-actions
github-actions Bot temporarily deployed to pull request July 28, 2024 06:04 Inactive
@github-actions
github-actions Bot temporarily deployed to pull request July 28, 2024 18:14 Inactive
@github-actions
github-actions Bot temporarily deployed to pull request August 2, 2024 17:07 Inactive
@github-actions
github-actions Bot temporarily deployed to pull request August 2, 2024 17:18 Inactive
@github-actions
github-actions Bot temporarily deployed to pull request August 2, 2024 21:19 Inactive
@github-actions
github-actions Bot temporarily deployed to pull request August 11, 2024 15:15 Inactive
@pjbull pjbull changed the title WIP: Minimal ADLS gen2 support Minimal ADLS gen2 support Aug 11, 2024

@pjbull pjbull left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A couple of typos/tweaks

Comment thread cloudpathlib/azure/azblobclient.py Outdated
Comment thread cloudpathlib/azure/azblobclient.py Outdated
Comment thread docs/docs/authentication.md Outdated
Comment thread tests/mock_clients/mock_azureblob.py
Comment thread tests/mock_clients/mock_azureblob.py Outdated
@pjbull

pjbull commented Aug 13, 2024 •

Copy link
Copy Markdown
Member Author

@M0dEx Thanks for your patience—ended up being more here than I expected. I think this is basically ready except a few small typos/tweaks. If you've got time, it would be great if you took this branch for a test drive and took a look at the code changes.

@pjbull
pjbull force-pushed the azure-data-lake-gen2 branch from c02cc47 to 1e09e0a Compare August 15, 2024 00:06
@github-actions
github-actions Bot temporarily deployed to pull request August 15, 2024 00:07 Inactive
@pjbull

pjbull commented Aug 20, 2024

Copy link
Copy Markdown
Member Author

Planning to rebase on top of #461 when that is in so that tests are reliable.

@pjbull pjbull changed the title Minimal ADLS gen2 support Initial ADLS gen2 support Aug 22, 2024
@pjbull
pjbull force-pushed the azure-data-lake-gen2 branch from 1e09e0a to 12f7388 Compare August 22, 2024 18:15
@github-actions
github-actions Bot temporarily deployed to pull request August 22, 2024 18:16 Inactive
@pjbull

pjbull commented Aug 22, 2024

Copy link
Copy Markdown
Member Author

@jayqi Should be ready for the release. Scope/maintenance burden is a little larger, but I think this is a large standing issue with a fair amount of interest so it is worth it to get in.

@jayqi jayqi left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay, it feels a little messy but I think it's reasonable.

Comment thread HISTORY.md Outdated
@github-actions
github-actions Bot temporarily deployed to pull request August 23, 2024 23:56 Inactive
@pjbull

pjbull commented Aug 24, 2024

Copy link
Copy Markdown
Member Author

Okay, it feels a little messy but I think it's reasonable.

Truer words were never spoken.

@github-actions
github-actions Bot temporarily deployed to pull request August 24, 2024 00:18 Inactive
@github-actions
github-actions Bot temporarily deployed to pull request August 28, 2024 04:30 Inactive
@github-actions
github-actions Bot temporarily deployed to pull request August 28, 2024 04:44 Inactive
@pjbull
pjbull merged commit 5656879 into master Aug 28, 2024
@pjbull
pjbull deleted the azure-data-lake-gen2 branch August 28, 2024 15:41

This branch was previously deployed

1 inactive deployment
pull request — 9853b6b1 Deployed Aug 28, 2024 by github-actions[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support Azure Data Lake Storage Gen2

3 participants