You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
inspect-trace: fall back to tool-result text when no span has error status #140
Is your feature request related to a problem? Please describe.
eval-author-inspect-trace reads failures with nemo intake spans list ... --filter.status=error (SKILL.md, "Read selective detail"). For traces from LangChain Deep Agents exported with NeMo Relay up to 0.10.4, every tool span is SUCCESS, even when the tool failed: Intake marks tool spans ERROR only for some producers (NVIDIA-NeMo/nemo-helix#1265), and Relay marked failed LangChain tool calls only from NVIDIA/NeMo-Relay#1416 (in 0.11.0-rc.3). So the filter returns nothing, and the skill concludes there were no errors. In our FreeCAD CAD agent's traces, 21% of shell calls had crashed the program with exit code 0.
Describe the solution you'd like
A fallback in the skill: if --filter.status=error returns no spans, scan tool results for failure text (Error, Traceback, non-zero exit codes, exit_code in the result) before concluding the run had no errors, and say which method was used.
Describe alternatives you've considered
Waiting for every producer to set span status, which leaves older traces unreadable.
Activity