Skip to content

inspect-trace: fall back to tool-result text when no span has error status #140

Open

Description

@phermosomore

Is your feature request related to a problem? Please describe.

eval-author-inspect-trace reads failures with nemo intake spans list ... --filter.status=error (SKILL.md, "Read selective detail"). For traces from LangChain Deep Agents exported with NeMo Relay up to 0.10.4, every tool span is SUCCESS, even when the tool failed: Intake marks tool spans ERROR only for some producers (NVIDIA-NeMo/nemo-helix#1265), and Relay marked failed LangChain tool calls only from NVIDIA/NeMo-Relay#1416 (in 0.11.0-rc.3). So the filter returns nothing, and the skill concludes there were no errors. In our FreeCAD CAD agent's traces, 21% of shell calls had crashed the program with exit code 0.

Describe the solution you'd like

A fallback in the skill: if --filter.status=error returns no spans, scan tool results for failure text (Error, Traceback, non-zero exit codes, exit_code in the result) before concluding the run had no errors, and say which method was used.

Describe alternatives you've considered

Waiting for every producer to set span status, which leaves older traces unreadable.

Additional context

Example: NVIDIA/nemoclaw-community#213

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions