[Feature] Add VLLM_CONNECTION to the ChatModel out-of-the-box #928

Activity

1. piosystems

   added

   feature[Issue Type] New features or improvements to existing features.[Issue Type] New features or improvements to existing features.

   priority/majorDefault priority of the PR or issue.Default priority of the PR or issue.

   on Jul 26, 2026
2. Zhuoxi2000 commented on Jul 27, 2026

   

   Zhuoxi2000

   on Jul 27, 2026

   Contributor

   More actions

   Thanks for raising this. I’d like to pick it up.

   Since vLLM is OpenAI-compatible and our existing OpenAI connections already support a custom api_base_url, I think the simplest path could be either:

   1. Add a vllm YAML alias pointing to the existing OpenAI-compatible setup, or
   2. Add a thin dedicated VLLM_CONNECTION with vLLM-friendly defaults, like a local default api_base_url and optional API key.

   I’d lean toward the dedicated connection for discoverability, but happy to go with whichever direction the maintainers prefer. Once the scope is confirmed, I can open a PR.

   👍React with 👍1Reacted by Pius Onobhayedo
3. piosystems commented on Jul 27, 2026

   

   piosystems

   on Jul 27, 2026

   Last edited by piosystems

   Author

   More actions

   Thanks for raising this. I’d like to pick it up.

   Since vLLM is OpenAI-compatible and our existing OpenAI connections already support a custom api_base_url, I think the simplest path could be either:

   1. Add a vllm YAML alias pointing to the existing OpenAI-compatible setup, or
   2. Add a thin dedicated VLLM_CONNECTION with vLLM-friendly defaults, like a local default api_base_url and optional API key.

   I’d lean toward the dedicated connection for discoverability, but happy to go with whichever direction the maintainers prefer. Once the scope is confirmed, I can open a PR.

   I would lean as well toward the dedicated VLLM connection, for discoverability.
4. Zhuoxi2000 commented on Jul 27, 2026

   

   Zhuoxi2000

   on Jul 27, 2026

   Contributor

   More actions

   Thanks @piosystems — agreed on going with a dedicated connection.

   The shape I’m planning is:

   • Java: add VLLMChatModelConnection / VLLMChatModelSetup under the existing integrations/chat-models/openai module, similar to how Azure OpenAI is packaged today. This would reuse the OpenAI-compatible client, default api_base_url to http://localhost:8000/v1, and make api_key optional.
   • Python: add flink_agents/integrations/chat_models/vllm/vllm_chat_model.py, mirroring the existing chat_models/azure layout.
   • Add ResourceName.ChatModel.VLLM_CONNECTION / VLLM_SETUP and the Python variants.
   • Add a vllm alias in both Aliases.java and aliases.py.
   • Add docs and a YAML example, following [docs][yaml] Polish yaml doc and add bedrock alias #812.

   I’ll start with this direction. If a committer would prefer to scope it down to the alias-only approach, just let me know and I can adjust the alias would be included either way.

   👍React with 👍2Reacted by Pius Onobhayedo and Wenjin Xie
5. Zhuoxi2000

   added a commit that references this issue on Aug 1, 2026

   [integration] Add vLLM chat model support for Java and Python

   ...

   This issue will close once commit 363eb4b is merged into the 'main' branch.

   363eb4b
6. wenjin272

   added

   fixVersion/0.4.0

   on Aug 12, 2026
7. wenjin272

   assigned

   Zhuoxi2000

   on Aug 12, 2026
8. wenjin272

   closed this as completedon Aug 12, 2026