feat(langchain-backend-aws): add S3Backend for Deep Agents - #1035
Open
5enxia (5enxia) wants to merge 8 commits into
Open
feat(langchain-backend-aws): add S3Backend for Deep Agents#10355enxia (5enxia) wants to merge 8 commits into
5enxia (5enxia) wants to merge 8 commits into
Conversation
Introduce a new `langchain-backend-aws` package providing an S3-backed implementation of the Deep Agents `BackendProtocol` (ls/read/write/edit/ glob/grep/upload_files/download_files). Key design points: - Files are stored as raw S3 objects (no envelope) so they remain directly readable from the S3 Console and downstream tooling. - Multi-layer SSRF guard on `endpoint_url` and `extra_boto_config.proxies` (scheme allow-list, CRLF/NUL rejection, IDNA + NFKC normalisation, loopback / link-local / RFC1918 / wildcard-DNS blocking, IPv4-mapped IPv6 and zone-id handling). Private endpoints require explicit `allow_private_endpoints=True` for LocalStack/MinIO. - Path-to-key translation rejects traversal, overlong UTF-8, double encoding, Windows drive letters and NFKC fold-equivalents; the bucket prefix shape is held invariant by `_assert_prefix_shape`. - Fail-closed knobs cap memory and CPU surface: `max_file_size_mb`, `grep_max_pattern_length`, `grep_max_line_length`, `grep_max_pattern_metachars`, `grep_regex_timeout`, `glob_max_pattern_length`, `glob_max_objects`, `ls_max_objects`, `download_concurrency`, `binary_read_mode`. - Optimistic concurrency via `PutObject` with `IfNoneMatch="*"` (write) and `IfMatch=<ETag>` (edit); credentials are never logged (`repr=False` on `S3BackendConfig`). - Boto client built with `retries=adaptive` and a deepcopy'd `extra_boto_config` so aliasing cannot mutate post-construction state. Tested with 39 unit modules (no network, `--disable-socket`) and 8 integration modules verified against MinIO; coverage spans SSRF, traversal, overlong encoding, ReDoS caps, concurrency, prefix violations, and protocol-signature parity with deepagents 0.5.6.
Register the new package in `.github/scripts/check_diff.py`, `_release.yml`, `integration_test.yml`, and `pr_lint.yml` so lint, unit/integration tests, and release workflows pick it up alongside the existing `langchain-aws` and `langgraph-checkpoint-aws` packages.
There was a problem hiding this comment.
Pull request overview
This PR introduces a new libs/langchain-backend-aws Python package that implements Deep Agents’ BackendProtocol using Amazon S3 object storage, with strong emphasis on tenant isolation (bucket+prefix), safety (path traversal + SSRF defenses), and operational guardrails (caps + logging). It also adds a comprehensive unit/integration test suite and wires the new library into existing CI/release workflows.
Changes:
- Add
langchain_backend_awspackage withS3Backend+S3BackendConfigand S3-backed implementations forls/read/write/edit/glob/grep/upload_files/download_files(sync + async delegation). - Add security and robustness features (SSRF validation, traversal hardening, prefix invariants, fail-closed semantics, size/object caps, logging/audit behaviors).
- Add extensive unit + integration tests and include the new library in PR lint, integration test, and release workflows.
Reviewed changes
Copilot reviewed 65 out of 71 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| libs/langchain-backend-aws/README.md | New package documentation, configuration, security notes, and test instructions. |
| libs/langchain-backend-aws/pyproject.toml | Defines the new package metadata, dependencies, and lint/test tooling config. |
| libs/langchain-backend-aws/Makefile | Adds per-package lint/test/integration/coverage targets for the new library. |
| libs/langchain-backend-aws/langchain_backend_aws/init.py | Top-level public re-exports for S3Backend and S3BackendConfig. |
| libs/langchain-backend-aws/langchain_backend_aws/s3/init.py | s3 subpackage public surface (__all__) and re-exports. |
| libs/langchain-backend-aws/langchain_backend_aws/s3/backend.py | Main BackendProtocol implementation backed by S3. |
| libs/langchain-backend-aws/langchain_backend_aws/s3/_config.py | S3BackendConfig validation and boto3 client construction helpers. |
| libs/langchain-backend-aws/langchain_backend_aws/s3/_defaults.py | Centralized defaults and allow-lists for config/caps. |
| libs/langchain-backend-aws/langchain_backend_aws/s3/_ssrf.py | SSRF guard implementation for endpoint_url and proxy URLs. |
| libs/langchain-backend-aws/langchain_backend_aws/s3/_paths.py | Path-to-key/key-to-path mapping and traversal/encoding hardening. |
| libs/langchain-backend-aws/langchain_backend_aws/s3/_internal.py | Shared internal helpers (caps, glob compilation, oversize handling, etc.). |
| libs/langchain-backend-aws/langchain_backend_aws/s3/_read.py | Extracted read implementation (pagination + binary handling). |
| libs/langchain-backend-aws/langchain_backend_aws/s3/_write.py | Extracted write/edit implementations (conditional PUT/ETag semantics). |
| libs/langchain-backend-aws/langchain_backend_aws/s3/_ls.py | Extracted ls implementation (delimiter listing + caps + fail-closed checks). |
| libs/langchain-backend-aws/langchain_backend_aws/s3/_glob.py | Extracted glob implementation (regex-based globbing + caps + fail-closed checks). |
| libs/langchain-backend-aws/langchain_backend_aws/s3/_grep.py | Extracted grep implementation (regex search + ReDoS guards + caps). |
| libs/langchain-backend-aws/langchain_backend_aws/s3/_io.py | Extracted upload_files/download_files implementations (caps + concurrency). |
| libs/langchain-backend-aws/tests/init.py | Marks the new test package root. |
| libs/langchain-backend-aws/tests/unit_tests/init.py | Marks the unit test package. |
| libs/langchain-backend-aws/tests/unit_tests/s3/init.py | Marks the S3 unit test package. |
| libs/langchain-backend-aws/tests/unit_tests/s3/_helpers.py | Shared test helpers for mocked S3 responses/clients. |
| libs/langchain-backend-aws/tests/unit_tests/s3/test_imports.py | Pins public __all__ exports for package stability. |
| libs/langchain-backend-aws/tests/unit_tests/s3/test_constructor.py | Constructor/config and async delegation tests. |
| libs/langchain-backend-aws/tests/unit_tests/s3/test_build_client.py | Tests boto3 client build + extra_boto_config handling. |
| libs/langchain-backend-aws/tests/unit_tests/s3/test_paths.py | Tests path mapping and traversal protection. |
| libs/langchain-backend-aws/tests/unit_tests/s3/test_prefix_traversal.py | Tests prefix traversal-segment rejection in config/backend creation. |
| libs/langchain-backend-aws/tests/unit_tests/s3/test_require_prefix.py | Tests require_prefix fail-closed semantics. |
| libs/langchain-backend-aws/tests/unit_tests/s3/test_prefix_shape_invariant.py | Pins prefix trailing-slash invariant for helper functions. |
| libs/langchain-backend-aws/tests/unit_tests/s3/test_path_to_key_symmetry.py | Defense-in-depth tests for prefix containment symmetry. |
| libs/langchain-backend-aws/tests/unit_tests/s3/test_double_encoded_traversal.py | Pins double-encoded traversal behavior in path_to_key. |
| libs/langchain-backend-aws/tests/unit_tests/s3/test_empty_key_defense.py | Ensures root path doesn’t map to empty key for file ops. |
| libs/langchain-backend-aws/tests/unit_tests/s3/test_endpoint_url_validation.py | SSRF/scheme/host validation tests for endpoint_url. |
| libs/langchain-backend-aws/tests/unit_tests/s3/test_wildcard_dns_ssrf.py | SSRF tests for wildcard-DNS suffix blocks (nip.io, etc.). |
| libs/langchain-backend-aws/tests/unit_tests/s3/test_proxies_ssrf.py | SSRF + shape validation tests for extra_boto_config['proxies']. |
| libs/langchain-backend-aws/tests/unit_tests/s3/test_extra_boto_immutability.py | Tests deep-copy immutability of extra_boto_config. |
| libs/langchain-backend-aws/tests/unit_tests/s3/test_review_followups.py | Regression tests for prior review findings (config validation, cache clear, etc.). |
| libs/langchain-backend-aws/tests/unit_tests/s3/test_review_round15.py | Additional regression tests for later review findings. |
| libs/langchain-backend-aws/tests/unit_tests/s3/test_protocol_signature.py | Pins edit signature against upstream protocol. |
| libs/langchain-backend-aws/tests/unit_tests/s3/test_ls.py | Unit tests for ls. |
| libs/langchain-backend-aws/tests/unit_tests/s3/test_glob.py | Unit tests for glob behaviors and caps. |
| libs/langchain-backend-aws/tests/unit_tests/s3/test_glob_double_star.py | Regression coverage for ** translation semantics. |
| libs/langchain-backend-aws/tests/unit_tests/s3/test_glob_pattern_length.py | Tests glob pattern length cap enforcement. |
| libs/langchain-backend-aws/tests/unit_tests/s3/test_glob_metachar_cap.py | Tests glob wildcard/metachar cap enforcement. |
| libs/langchain-backend-aws/tests/unit_tests/s3/test_prefix_violation.py | Ensures fail-closed on out-of-prefix paginator leaks. |
| libs/langchain-backend-aws/tests/unit_tests/s3/test_grep.py | Unit tests for grep matching, guards, and fail-closed semantics. |
| libs/langchain-backend-aws/tests/unit_tests/s3/test_grep_robustness.py | Robustness tests for malformed S3 responses in grep flow. |
| libs/langchain-backend-aws/tests/unit_tests/s3/test_grep_metachar_cap.py | Tests grep metacharacter cap guard. |
| libs/langchain-backend-aws/tests/unit_tests/s3/test_read.py | Unit tests for read pagination, binary handling, and oversize defenses. |
| libs/langchain-backend-aws/tests/unit_tests/s3/test_read_binary_mode.py | Tests binary_read_mode behaviors. |
| libs/langchain-backend-aws/tests/unit_tests/s3/test_read_capped_close.py | Ensures streaming bodies are closed on all capped-read exit paths. |
| libs/langchain-backend-aws/tests/unit_tests/s3/test_write.py | Unit tests for write and conditional-create semantics. |
| libs/langchain-backend-aws/tests/unit_tests/s3/test_edit.py | Unit tests for edit and conditional-update semantics. |
| libs/langchain-backend-aws/tests/unit_tests/s3/test_upload_files.py | Unit tests for upload_files, including oversize handling. |
| libs/langchain-backend-aws/tests/unit_tests/s3/test_download_files.py | Unit tests for download_files mapping, logging, and oversize handling. |
| libs/langchain-backend-aws/tests/unit_tests/s3/test_download_robustness.py | Robustness tests for per-path exception handling in downloads. |
| libs/langchain-backend-aws/tests/unit_tests/s3/test_download_concurrency.py | Unit tests for parallel vs sequential download behavior. |
| libs/langchain-backend-aws/tests/unit_tests/s3/test_download_base_exception.py | Regression tests around BaseException-like teardown handling. |
| libs/langchain-backend-aws/tests/integration_tests/init.py | Marks the integration test package. |
| libs/langchain-backend-aws/tests/integration_tests/s3/init.py | Marks the S3 integration test package. |
| libs/langchain-backend-aws/tests/integration_tests/s3/conftest.py | Shared fixtures/env gating for live-store integration tests. |
| libs/langchain-backend-aws/tests/integration_tests/s3/test_backend.py | End-to-end integration tests against S3-compatible stores. |
| libs/langchain-backend-aws/tests/integration_tests/s3/test_compile.py | Placeholder compile/import smoke test. |
| libs/langchain-backend-aws/tests/integration_tests/s3/test_download_concurrency.py | Live-store integration tests for download concurrency. |
| libs/langchain-backend-aws/tests/integration_tests/s3/test_grep_metachar_cap.py | Live-store integration test for grep metachar cap. |
| libs/langchain-backend-aws/tests/integration_tests/s3/test_read_binary_mode.py | Live-store integration tests for binary_read_mode. |
| libs/langchain-backend-aws/tests/integration_tests/s3/test_require_prefix.py | Live-store integration test for require_prefix. |
| .github/workflows/pr_lint.yml | Adds the new library scope to PR lint workflow allowlist. |
| .github/workflows/integration_test.yml | Includes new library path in integration-test workflow triggers. |
| .github/workflows/_release.yml | Includes new library path in release workflow triggers. |
| .github/scripts/check_diff.py | Adds new library directory so CI selection logic recognizes it. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- pass MagicMock client to S3Backend in unit tests so pytest-socket's --disable-socket does not trip the implicit boto3.client construction - patch boto3.client in the build_client warning test for the same reason - skip the integration autouse ensure_bucket fixture when AWS credentials are absent so test_compile (import-only) does not require credentials - type integration test/conftest s3_client fixtures as Any; boto3.client is a factory function and mypy rejects it as a type annotation - apply ruff format to test_review_followups.py and test_review_round15.py
test_review_followups.py / test_review_round15.py bundled unrelated classes under review-process names. Split each class into a file named for its subject so the suite is navigable by topic: - test_extra_boto_config_validation.py - test_proxies_config.py - test_download_files_teardown.py - test_read_offset_limit_warning.py - test_read_capped_body_type.py - test_glob_cache.py
- README: drop user_agent from extra_boto_config allowlist to match
the actual ALLOWED_BOTO_KEYS (full override is intentionally rejected)
- README: document that oversize is surfaced via the OVERSIZE_ERROR_TAG
("oversize"), not permission_denied
- README: correct glob regex cache as per-instance (built by
make_glob_compiler) rather than process-global
- _io.py: fix docstring claim that asyncio.CancelledError and
concurrent.futures.CancelledError are the same class on 3.8+; name
both explicitly in the teardown except clause so a future-cancelled
path is also re-raised after draining
- _ls.py / _glob.py: add fail-closed guard for malformed paginator
responses (KeyError/TypeError on obj["Key"]/obj["LastModified"]/
prefix_entry["Prefix"]) so a misbehaving stub or S3-compatible store
cannot raise across the BackendProtocol boundary
- tests: rename and reframe test_download_base_exception to reflect
that concurrent.futures.CancelledError is an Exception subclass;
add an asyncio.CancelledError re-raise test to exercise the true
BaseException teardown path; add test_ls_robustness covering the
new fail-closed guards in ls/glob
Merge 31 small per-concern test files into the 7 topic owners (constructor, paths, read, ls, glob, grep, download_files) to shrink the PR's changed-files footprint while keeping every assertion. - read: fold binary_read_mode, read_capped_object close-paths/non-bytes body defense, and the offset/limit base64 warning into test_read.py - glob: fold clear_glob_cache, `**` translation, and the pattern length / wildcard metachar caps into test_glob.py - grep: fold the metachar cap and malformed-response robustness into test_grep.py - ls: fold malformed-listing robustness into test_ls.py - download_files: fold concurrency, robustness, teardown signals, and teardown logging into test_download_files.py - paths: fold path/key symmetry, double-encoded traversal, prefix shape invariant, prefix traversal at construction, require_prefix, empty key defense, and the prefix-violation fail-closed contract into test_paths.py - constructor: fold build_client, endpoint_url / wildcard-DNS / proxies SSRF, proxies_config allow-list + audit logging, extra_boto_config validation, and extra_boto_config immutability into test_constructor.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
issue: #915
Summary
Adds a new
langchain-backend-awspackage underlibs/that implements the Deep AgentsBackendProtocolon top of Amazon S3. Files are stored as S3 objects so they remain directly readable from the S3 Console and downstream tooling.Features
ls,read,write,edit,glob,grep,upload_files,download_files(sync + async) implemented with signatures matching Deep Agents 0.5.x exactly.prefixis held invariant by the path-to-key layer.writeusesPutObjectwithIfNoneMatch="*"(atomic create);editusesIfMatch=<ETag>(conflict-aware update).max_file_size_mb,download_concurrency,glob_max_objects,ls_max_objects,grep_max_objects,grep_regex_timeout, etc.Test plan
make tests(39 modules, socket disabled)make integration_testsagainst MinIO (21 tests)BackendProtocolconformance verified againstdeepagents0.5.6AI involvement disclaimer
This PR was authored with the assistance of Claude Code (Anthropic). All design decisions and code changes have been reviewed and approved by the human author.