Skip to content

feat: pass through all backend-supported sampling/serving parameters #151

Closed

Description

@ZuyiZhou

Problem

Raven forwards only a fixed subset of sampling parameters to the serving backend. Anything outside that subset is silently dropped, even when the backend supports it. Two incidents where this bit us:

  1. repetition_penalty — sglang supports it, but there is no way to get it through Raven's model config. We had to interpose a local proxy shim on the request path just to set one sampling knob.
  2. enable_thinking: false (Qwen3-style chat-template switch, normally passed via extra_body / chat_template_kwargs) — unreachable through Raven config; required patching the package.

Both are legitimate, backend-documented parameters; needing a fork for each new knob doesn't scale.

Proposal

A passthrough dict on the model/backend config, forwarded verbatim into the request payload (merged into the OpenAI-compatible body / extra_body):

model:
  sampling_extra:            # merged into sampling params
    repetition_penalty: 1.05
  request_extra:             # merged into extra_body
    chat_template_kwargs: { enable_thinking: false }

Exact shape is bikesheddable — the ask is: anything the backend accepts should be reachable from config without a fork.

Activity

  1. added a commit that references this issue on Aug 10, 2026
    56f7095
  2. added a commit that references this issue on Sep 14, 2026
    5719cbe
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions