Skip to content

feat(aws): native Bedrock batch inference support - #1117

Draft
Islam Elkadi (islamelkadi) wants to merge 1 commit into
langchain-ai:mainfrom
islamelkadi:feat/bedrock-batch-inference
Draft

feat(aws): native Bedrock batch inference support#1117
Islam Elkadi (islamelkadi) wants to merge 1 commit into
langchain-ai:mainfrom
islamelkadi:feat/bedrock-batch-inference

Conversation

@islamelkadi

@islamelkadi Islam Elkadi (islamelkadi) commented Jun 22, 2026

Copy link
Copy Markdown
Contributor

Summary

A design-preview skeleton for native Amazon Bedrock batch inference using the Converse API format, as proposed in issue #1116 .

This PR sketches the public surface that would close that gap. Every method body is raise NotImplementedError — the goal is to align on shape, naming, and placement with maintainers before implementing.

What's included

  • langchain_aws/batch/_formatting.py — internal LangChain ⇄ Converse JSONL conversion + client-side validation, reusing _messages_to_bedrock / _parse_response from bedrock_converse.py.
  • langchain_aws/batch/manager.pyBedrockBatchManager (plain orchestration class) and BatchJob (pydantic handle). Lifecycle: submit / get_status / get_results / cancel.
  • langchain_aws/batch/node.py — optional LangGraph BedrockBatchNode (submit → checkpoint via interrupt → resume → results).
  • Public API wiring in langchain_aws/__init__.py and a new optional batch extra (langgraph) in pyproject.toml / uv.lock.

Design decisions (seeking feedback)

  • BedrockBatchManager is a plain class, not a pydantic model — it's an imperative orchestrator (not a Runnable), so it mirrors AmazonS3Vectors: clients built in __init__ via create_aws_client, with optional client injection for testing.
  • Credentials live in one place — region/credentials are reused from the passed-in ChatBedrockConverse rather than re-declaring the credential block.
  • BatchJob stays pydantic and is threaded through the lifecycle methods (not just returned), so it serializes into LangGraph state and carries the context (output_s3_uri, record_count) needed to locate/validate results.
  • LangGraph is an optional dependency — the heavy import is deferred into BedrockBatchNode, so import langchain_aws.batch stays light and raises a friendly install hint only on construction.
  • Pre-flight validation is the UX centerpiece: submit() fails fast in four stages (client-side validation → S3 upload → CreateModelInvocationJob → a brief infra-verification poll) so a misconfigured job surfaces in seconds, not hours.

Intentionally NOT in this PR

  • Method implementations (all raise NotImplementedError).
  • Tests — will follow the implementation, with VCR cassettes for the AWS shapes per the repo convention.

Areas that need careful review

  • First plain s3 client + first S3 object I/O in the package. Repo-wide there is no existing "s3" client or S3 helper (only an s3vectors client), so the credential path, s3:// parsing, and output-file discovery in _upload_jsonl / _iter_output_lines are net-new surface area.
  • API ergonomics: lifecycle methods take a BatchJob (strict) rather than a raw ARN string — is that the right contract, or should they accept str | BatchJob?
  • langgraph>=1.0.0 lower bound for the batch extra.
  • submit() blocking for verify_seconds (default 30s) — acceptable, or should verification be opt-in / async?

Additional food for thought

Likely out of scope due to the potential requirement of external infrastructure:

  • Suppose a user submits a massive job that will take 20+ hours to completely process. In a remote LangGraph environment, the node doesn't necessarily have to be awake this whole time but will require some external mechanism to wake up that node. Perhaps that external integration could be an EventBride rule + HTTPS endpoint as a target (assuming the node is exposed as a Lambda / ECS / EC2 / etc endpoint)? These endpoints will surely require an API Key, OAuth, or some form of auth to secure it.

That being said, in general does this direction align with the roadmap? Happy to iterate on the design here before filling in the implementation.

Skeleton (signatures + docstrings only) proposing BedrockBatchManager and an optional LangGraph BedrockBatchNode for native Bedrock batch inference via the Converse API format. Method bodies raise NotImplementedError to paint a design picture for maintainers.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant