Repository navigation
Seperate api_server and pytorch engine into different processors - #3627
Merged
Merged
Conversation
added 14 commits
June 8, 2025 19:36
lvhan028
changed the title
lvhan028
reviewed
Jun 26, 2025
lvhan028
reviewed
Jun 26, 2025
lvhan028
reviewed
Jun 26, 2025
| logger.info(f'create server with url={url}') | ||
| logger.info(f'backend_config_dp={backend_config_dp} gpus={gpu_ids_per_dp}') | ||
| proc = mp.Process(target=_run_server, args=(gpu_ids_per_dp, model_path), kwargs=cur_server_kwargs, daemon=True) | ||
| proc = mp.Process(target=_run_server, args=(gpu_ids_per_dp, model_path), kwargs=cur_server_kwargs) |
Collaborator
There was a problem hiding this comment.
What's the motivation to remove the "daemon" indicator?
If we start the api_server during rollout, do we need to remove it also if MP is enabled?
Collaborator
Author
There was a problem hiding this comment.
spawn process in daemon process is not allowed.
lvhan028
reviewed
Jun 26, 2025
lvhan028
reviewed
Jun 26, 2025
|
|
||
| handle_torchrun() | ||
| _, pipeline_class = get_task(model_path) | ||
| if isinstance(backend_config, PytorchEngineConfig): |
Collaborator
There was a problem hiding this comment.
It is noticed that enable_mp_engine defaults to False, but api_server changes it to True by force. So, we don't recommend it when pipeline is used or pipeline hasn't support this feature yet
Collaborator
Author
There was a problem hiding this comment.
Host performance is acceptable on pipeline. I don't want the system to be too complex.
added 2 commits
June 26, 2025 14:25
oliveagle
pushed a commit
to oliveagle/lmdeploy
that referenced
this pull request
May 22, 2026
…ernLM#3627) * mp router * optimize * better streaming * optimize streaming * close engine * safe exit * support pd * fix loader * optimize * safe exit * safe exit * fix comment * remove empty line
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The PR adds a new layer between API server and pytorch engine, so both of them can be run in their own process.