Skip to content

bug: context-limit salvage silently promotes think fragments into final_answer; think-tag misalignment on backends without a reasoning parser #152

Closed

Description

@ZuyiZhou

Two related integrity gaps in the think channel, filed together because they break the same contract: downstream consumers should be able to trust that final_answer is an actual answer and that think markup is well-formed.

1. Salvage promotes think fragments into final_answer with no marker

When a run hits the context limit, the salvage path can take a truncated reasoning fragment and emit it as final_answer. The result is an "answer" containing an orphan </think> (or trailing off mid-reasoning), indistinguishable from a genuine answer by any flag in the payload. Consumers only discover this by scanning answer text for stray think markup — we lost a full data-cleaning pass to this before knowing to look.

Proposal: salvage output must be marked (is_salvaged: true, or finish_reason: context_limit_salvaged); a think fragment should never be promoted to final_answer silently. Ideally the salvaged text goes into a separate field and final_answer stays null.

2. Think-tag misalignment when the backend has no reasoning parser

When the serving backend is launched without a reasoning parser (e.g. sglang without --reasoning-parser), the chat template emits the opening <think> tag inside the generation prompt, so the completion contains only bare reasoning text plus a closing </think>. Raven records this as-is: trajectories carry a broken protocol (closing tag with no opening tag), and every consumer ends up hand-writing the same canonicalization.

The root cause is arguably serving-side configuration — but Raven is the layer that actually sees the malformed stream, and is best positioned to either normalize it (re-attach the opening tag) or at minimum emit a loud warning that the backend's reasoning parsing looks misconfigured.

Proposal: detect unbalanced think tags in assistant messages at ingestion time; normalize, or warn loudly.

Activity

  1. added a commit that references this issue on Aug 10, 2026
    56f7095
  2. added a commit that references this issue on Sep 14, 2026
    5719cbe
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions