[Bug] Metrics endpoint 404

Checklist

1. I have searched related issues but cannot get the expected help.
2. The bug has not been fixed in the latest version.
3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.

Describe the bug

lmdeploy-server | HINT: Please open http://0.0.0.0:23333 in a browser for detailed api usage!!!
lmdeploy-server | HINT: Please open http://0.0.0.0:23333 in a browser for detailed api usage!!!
lmdeploy-server | HINT: Please open http://0.0.0.0:23333 in a browser for detailed api usage!!!
lmdeploy-server | INFO: Started server process [1]
lmdeploy-server | INFO: Waiting for application startup.
lmdeploy-server | INFO: Application startup complete.
lmdeploy-server | INFO: Uvicorn running on http://0.0.0.0:23333 (Press CTRL+C to quit)
lmdeploy-server | INFO: 172.29.8.12:43056 - "GET /metrics HTTP/1.1" 404 Not Found
lmdeploy-server | INFO: 172.29.8.12:43056 - "GET /metrics HTTP/1.1" 404 Not Found
lmdeploy-server | INFO: 172.29.8.12:43056 - "GET /metrics HTTP/1.1" 404 Not Found
lmdeploy-server | INFO: 172.29.8.12:43056 - "GET /metrics HTTP/1.1" 404 Not Found
lmdeploy-server | INFO: 172.29.8.12:43056 - "GET /metrics HTTP/1.1" 404 Not Found

Reproduction

docker run --gpus all \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HUGGING_FACE_HUB_TOKEN="hf_f..."" \
    -p 23333:23333 \
    --ipc=host \
    openmmlab/lmdeploy:latest \
    lmdeploy serve api_server stelterlab/DeepSeek-R1-0528-Qwen3-8B-AWQ --enable-metrics

usage: lmdeploy [-h] [-v] {lite,serve,convert,list,check_env,chat} ...
lmdeploy: error: unrecognized arguments: --enable-metrics

Environment

(.venv) [dan@QNAP lmdeploy]$  docker run --gpus all \
>     -v ~/.cache/huggingface:/root/.cache/huggingface \
>     -p 23333:23333 \
>     --ipc=host \
>     openmmlab/lmdeploy:latest \
>     lmdeploy check_env

==========
== CUDA ==
==========

CUDA Version 12.4.1

Container image Copyright (c) 2016-2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.

This container image and its contents are governed by the NVIDIA Deep Learning Container License.
By pulling and using the container, you accept the terms and conditions of this license:
https://developer.nvidia.com/ngc/nvidia-deep-learning-container-license

A copy of this license is made available in this container at /NGC-DL-CONTAINER-LICENSE for your convenience.

sys.platform: linux
Python: 3.10.12 (main, May 27 2025, 17:12:29) [GCC 11.4.0]
CUDA available: True
MUSA available: False
numpy_random_seed: 2147483648
GPU 0,1: NVIDIA RTX A2000 12GB
GPU 2: NVIDIA GeForce GTX 1080
CUDA_HOME: /usr/local/cuda
NVCC: Cuda compilation tools, release 12.4, V12.4.131
GCC: x86_64-linux-gnu-gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
PyTorch: 2.7.1+cu126
PyTorch compiling details: PyTorch built with:
  - GCC 11.2
  - C++ Version: 201703
  - Intel(R) oneAPI Math Kernel Library Version 2024.2-Product Build 20240605 for Intel(R) 64 architecture applications
  - Intel(R) MKL-DNN v3.7.1 (Git Hash 8d263e693366ef8db40acc569cc7d8edf644556d)
  - OpenMP 201511 (a.k.a. OpenMP 4.5)
  - LAPACK is enabled (usually provided by MKL)
  - NNPACK is enabled
  - CPU capability usage: AVX2
  - CUDA Runtime 12.6
  - NVCC architecture flags: -gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_90,code=sm_90
  - CuDNN 90.5.1
  - Magma 2.6.1
  - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, COMMIT_SHA=e2d141dbde55c2a4370fac5165b0561b6af4798b, CUDA_VERSION=12.6, CUDNN_VERSION=9.5.1, CXX_COMPILER=/opt/rh/gcc-toolset-11/root/usr/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=1 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DLIBKINETO_NOXPUPTI=ON -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=range-loop-construct -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-unknown-pragmas -Wno-unused-parameter -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, TORCH_VERSION=2.7.1, USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=1, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF,

TorchVision: 0.22.1+cu126
LMDeploy: 0.9.1+db1849b
transformers: 4.53.1
gradio: 5.35.0
fastapi: 0.115.14
pydantic: 2.11.7
triton: 3.3.1
NVIDIA Topology:
	GPU0	GPU1	GPU2	CPU Affinity	NUMA Affinity	GPU NUMA ID
GPU0	 X 	PHB	PHB	0-7	0		N/A
GPU1	PHB	 X 	PHB	0-7	0		N/A
GPU2	PHB	PHB	 X 	0-7	0		N/A

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

Error traceback

Note:
I have also tried building this myself in a Dockerfile:

# Use the NVIDIA CUDA 12.4.1 development image as the base.
# This ensures all necessary CUDA libraries and tools are available.
FROM nvidia/cuda:12.4.1-devel-ubuntu22.04

# Install base dependencies like Python and pip.
RUN apt-get update && \
    apt-get install -y --no-install-recommends \
        python3 python3-pip python3-dev python-is-python3 && \
    rm -rf /var/lib/apt/lists/*

# Step 1: Install the core lmdeploy library specifically for CUDA 12.4.
RUN python3 -m pip install lmdeploy

# Set the working directory for the container.
WORKDIR /root

# Set the default command to start the LMDeploy OpenAI compatible API server.
# The actual model and parameters will be passed from docker-compose.
ENTRYPOINT ["lmdeploy", "serve", "api_server" ]

version: '3'
services:
  lmdeploy:
    build:
      context: .
      dockerfile: Dockerfile.lmdeploy
    image: lmdeploy-server:dan
    container_name: lmdeploy-server
    ipc: host
    restart: unless-stopped
    ports:
      - "23333:23333"
    volumes:
      - ~/.cache/huggingface:/root/.cache/huggingface
    environment:
      HUGGING_FACE_HUB_TOKEN: "hf_...."
      CUDA_DEVICE_ORDER: "PCI_BUS_ID"
      NVIDIA_VISIBLE_DEVICES: "0,2"
      CUDA_VISIBLE_DEVICES: "0,2"
      PYTORCH_CUDA_ALLOC_CONF: "max_split_size_mb:64,expandable_segments:True"
    command: >
      stelterlab/DeepSeek-R1-0528-Qwen3-8B-AWQ
      --tp 2
      --session-len 65536
      --cache-max-entry-count 0.92
      --quant-policy 8
      --rope-scaling-factor 1.0
      --log-level INFO
      --enable-metrics
    networks:
      - ragflow
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

Actually I notice this is installing LMDeploy: 0.9.1+db1849b and metrics was only released in 0.9.2 - not sure why its not fetching the latest version but even on this version it is still not working:

(.venv) [dan@QNAP lmdeploy]$   docker run --gpus all \
>     -v ~/.cache/huggingface:/root/.cache/huggingface \
>     -p 23333:23333 \
>     --ipc=host \
>     lmdeploy-server:dan \
>     check_env
sys.platform: linux
Python: 3.10.12 (main, May 27 2025, 17:12:29) [GCC 11.4.0]
CUDA available: True
MUSA available: False
numpy_random_seed: 2147483648
GPU 0,1: NVIDIA RTX A2000 12GB
GPU 2: NVIDIA GeForce GTX 1080
CUDA_HOME: /usr/local/cuda
NVCC: Cuda compilation tools, release 12.4, V12.4.131
GCC: x86_64-linux-gnu-gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
PyTorch: 2.7.1+cu126
PyTorch compiling details: PyTorch built with:
  - GCC 11.2
  - C++ Version: 201703
  - Intel(R) oneAPI Math Kernel Library Version 2024.2-Product Build 20240605 for Intel(R) 64 architecture applications
  - Intel(R) MKL-DNN v3.7.1 (Git Hash 8d263e693366ef8db40acc569cc7d8edf644556d)
  - OpenMP 201511 (a.k.a. OpenMP 4.5)
  - LAPACK is enabled (usually provided by MKL)
  - NNPACK is enabled
  - CPU capability usage: AVX2
  - CUDA Runtime 12.6
  - NVCC architecture flags: -gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_90,code=sm_90
  - CuDNN 90.5.1
  - Magma 2.6.1
  - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, COMMIT_SHA=e2d141dbde55c2a4370fac5165b0561b6af4798b, CUDA_VERSION=12.6, CUDNN_VERSION=9.5.1, CXX_COMPILER=/opt/rh/gcc-toolset-11/root/usr/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=1 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DLIBKINETO_NOXPUPTI=ON -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=range-loop-construct -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-unknown-pragmas -Wno-unused-parameter -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, TORCH_VERSION=2.7.1, USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=1, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF,

TorchVision: 0.22.1+cu126
LMDeploy: 0.9.2+unknown
transformers: 4.54.1
gradio: Not Found
fastapi: 0.116.1
pydantic: 2.11.7
triton: 3.3.1
NVIDIA Topology:
	GPU0	GPU1	GPU2	CPU Affinity	NUMA Affinity	GPU NUMA ID
GPU0	 X 	PHB	PHB	0-7	0		N/A
GPU1	PHB	 X 	PHB	0-7	0		N/A
GPU2	PHB	PHB	 X 	0-7	0		N/A

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

lmdeploy-server  | 2025-08-05 09:01:33,427 - lmdeploy - INFO - async_engine.py:290 - updated backend_config=TurbomindEngineConfig(dtype='auto', model_format='awq', tp=2, dp=1, device_num=2, attn_tp_size=2, attn_dp_size=1, mlp_tp_size=2, mlp_dp_size=1, outer_dp_size=1, session_len=65536, max_batch_size=128, cache_max_entry_count=0.92, cache_chunk_size=-1, cache_block_seq_len=64, enable_prefix_caching=False, quant_policy=8, rope_scaling_factor=1.0, use_logn_attn=False, download_dir=None, revision=None, max_prefill_token_num=8192, num_tokens_per_iter=8192, max_prefill_iters=8, devices=[0, 1], empty_init=False, communicator='nccl', hf_overrides=None)
lmdeploy-server  | HINT:    Please open http://0.0.0.0:23333 in a browser for detailed api usage!!!
lmdeploy-server  | HINT:    Please open http://0.0.0.0:23333 in a browser for detailed api usage!!!
lmdeploy-server  | HINT:    Please open http://0.0.0.0:23333 in a browser for detailed api usage!!!
lmdeploy-server  | INFO:     Started server process [1]
lmdeploy-server  | INFO:     Waiting for application startup.
lmdeploy-server  | INFO:     Application startup complete.
lmdeploy-server  | INFO:     Uvicorn running on http://0.0.0.0:23333 (Press CTRL+C to quit)
lmdeploy-server  | INFO:     172.29.8.13:38946 - "GET /metrics HTTP/1.1" 404 Not Found

I have also tried pulling down the latest docker image and running directly from that: docker pull openmmlab/lmdeploy:v0.9.2-cu12 and it is still not working

Replied in

[Feature] metrics support #3534

Currently, Turbomind (the default backend in the Docker image) does not support the metrics feature, which will be available soon.
For now, you can

set the --backend pytorch if you want to use the metrics system, or
manually apply the Trubomind metrics PR (it hasn't been merged into the main branch since we need some time to review the code, but the functionalities work well).

Yep I saw this in the other side - pytorch is incredibly slow and Turbomind seems incredibly fast - I would just like to see how fast, I am trying some external vllm benchmarks and they seem a bit unrealistic for my hardware.

I'll apply the PR tomorrow evening and see how it goes - thanks!

@voycey

Hi, @voycey
Can this PR do the favor? #3811

This issue is marked as stale because it has been marked as invalid or awaiting response for 7 days without any further response. It will be closed in 5 days if the stale label is not removed or if there is no further response.

I believe there is a PR to solve this now?

@voycey

I believe there is a PR to solve this now?

@voycey Hi, #3811 metrics PR for turbomind has been merged into the main branch. You may install the latest code and use --enable-metrics to turn on the feature when launching the API server.

This issue is marked as stale because it has been marked as invalid or awaiting response for 7 days without any further response. It will be closed in 5 days if the stale label is not removed or if there is no further response.

This issue is closed because it has been stale for 5 days. Please open a new issue if you have similar issues or you have any new updates now.

Navigation Menu

Repository navigation

[Bug] Metrics endpoint 404 #3815

Checklist

Describe the bug

Reproduction

Environment

Error traceback

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[Bug] Metrics endpoint 404 #3815

Description

Checklist

Describe the bug

Reproduction

Environment

Error traceback

Activity

voycey commented on Aug 5, 2025

voycey commented on Aug 5, 2025

voycey commented on Aug 5, 2025

CUHKSZzxy commented on Aug 5, 2025

voycey commented on Aug 5, 2025

lvhan028 commented on Aug 6, 2025

github-actions commented on Aug 19, 2025

voycey commented on Aug 19, 2025

CUHKSZzxy commented on Aug 19, 2025

github-actions commented on Aug 28, 2025

github-actions commented on Sep 3, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions