Skip to content

Server: support models unloading - #197

Merged
0xShug0 merged 3 commits into
0xShug0:mainfrom
nikich340:main
Aug 11, 2026
Merged

0xShug0 merged 3 commits into
0xShug0:mainfrom
nikich340:main

Conversation

@nikich340

@nikich340 nikich340 commented Aug 8, 2026 •

Copy link
Copy Markdown
Contributor

Summary
Adds two endpoints to server: v1/tasks/unload_models, v1/tasks/unload_all_models.

Purpose:
In some situations, I need to free all the possible VRAM while keeping server functional and ready for new tasks without restart. That's where these features will help.

Confirmation:
Successfully built and tested on Windows 11 (CUDA sm_120 avx-512 release build) with omnivoice, qwen3-asr models. The server correctly handles new endpoints. When a model is in use, it properly waits until it finishes and then do unloading. The next requests to server work fine, with model loading back to backend device.

@0xShug0

0xShug0 commented Aug 8, 2026

Copy link
Copy Markdown
Owner

@nikich340 Thanks for the PR! Two minor issues:

  1. LoadedModel::unload() resets model before session. Normal member destruction order would destroy session before model, and unload should preserve that safer lifetime order:
    offline=nullptr, streaming=nullptr, session.reset(), then model.reset().

  2. Typo: app/server/README.md: docs say model-ids, but code requires model_ids.

Could you fix them?

@nikich340

Copy link
Copy Markdown
Contributor Author

Sure, you are right, done!

@0xShug0
0xShug0 merged commit 9266f8f into 0xShug0:main Aug 11, 2026
6 checks passed
@0xShug0

0xShug0 commented Aug 11, 2026

Copy link
Copy Markdown
Owner

@nikich340 Merged. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants