Skip to content

Fix #66: Recover from malformed tool calls and stop after repeated format errors - #75

Merged
vicgupta merged 1 commit into
mainfrom
amf/issue-66
Oct 25, 2026
Merged

vicgupta merged 1 commit into
mainfrom
amf/issue-66

Conversation

@vicgupta

Copy link
Copy Markdown
Owner

Summary

  • New Executor.FormatError check before dispatch: a call is a format error when its arguments are invalid JSON / not an object, the tool name isn't in the run's catalog, a schema required field is missing (or null), or a field has the wrong JSON type. Applies to MCP tool schemas too.
  • Malformed calls are not executed (and no hooks run); the model gets format error: <reason>. Call one of: <tools> with arguments matching its schema. (capped at 500 chars). Well-formed calls in the same response run as before.
  • A step whose calls are all malformed increments a consecutive counter; any well-formed call resets it. At N (default 3) the run ends with new status RepeatedFormatError, empty submission, non-zero exit; limits are still checked first.
  • Config max_format_errors and flag --max-format-errors (flag overrides config). Must be an integer >= 1.
  • Visibility: tool.result gets "format_error": true; result event gets format_errors; summary line shows N format errors (+ last reason on RepeatedFormatError); trajectory gets format_errors, consecutive_format_errors, format_error_messages (message indexes), last_format_error. Original arguments are kept in the trajectory.
  • Resume: counter persists and is restored; checkpoint validation now accepts malformed calls only when their result is a marked format-error correction; RepeatedFormatError checkpoints are resumable like limit statuses.
  • README: flag table, config example, Exit status, new "Malformed tool calls" section.

Decisions on open questions

  1. No-tool-call responses stay Completed (unchanged).
  2. 0 is rejected (exit 2 for the flag); no "no cap" mode.
  3. Truncation (finish_reason=length) detection not added.
  4. Type validation is JSON types only (no enum/min/max). Integers still accept numeric strings, as the tools always did.
  5. No schema_version bump; fields are additive/omitempty and ToolSignature is unchanged, so existing checkpoints still resume.
  6. Names as proposed: RepeatedFormatError, max_format_errors / --max-format-errors, format_error.

Other choices:

  • In --read-only, calling a non-read-only built-in keeps the existing read-only denial (not a format error); a truly unknown tool is a format error (one test expectation updated accordingly).
  • Invalid config values (0, -1, abc, 1.5) fail before any provider request with a message naming max_format_errors, but exit 1 like every other config error (AC8 asked for 2; flags do exit 2). 1.5 is explicitly rejected (yaml would otherwise truncate it to 1).
  • Help shows integer defaults unquoted ((default 3)); usage golden updated for the wider flag column.

Tests

internal/agent/format_test.go covers AC1–AC7, AC9, AC11 (trajectory markers, resume with consecutive=2 → stops on next malformed step, unmarked malformed call rejected by validation), plus wrong-type and mixed-step reset. cmd/tinyagent/format_test.go covers AC5/AC7 end-to-end via a fake OpenAI server (default 3, flag 1, config 5, flag overrides config), AC8 (flag/config validation, zero provider requests), AC10 (--json stays pure JSONL, format_error and result.status/format_errors). Existing suite unchanged otherwise (AC12). gofmt/vet/build/test pass locally.

Closes #66

…rs (#66)

Classify malformed tool calls (invalid/non-object JSON arguments, unknown
tool, missing required field, wrong JSON type) as format errors: skip the
tool, return a short uniform correction, and end the run with
RepeatedFormatError after N consecutive all-malformed steps (default 3,
config max_format_errors, flag --max-format-errors). Record counts and
markers in the trajectory, --json events and summary; persist the counter
across --resume.
@vicgupta

Copy link
Copy Markdown
Owner Author

Rebased onto main by the hourly merge job (head now 051ebe1). Two notes on the rebase:

  1. Beyond conflict resolution: removed the duplicate captureStdout helper from cmd/tinyagent/format_test.go — it is identical to the one in cmd/tinyagent/schema_test.go (landed on main via Add --output-schema to end runs with a validated, schema-conforming JSON result for scripting and CI #72), and keeping both failed go vet with a redeclaration error. Main's copy is kept.

  2. The internal/agent/loop.go conflict (main refactored the tool-call loop into toolStep/recordToolResult helpers) was resolved by keeping those helpers: well-formed calls still go through toolStep, while format-error calls are handled in the loop body, still skip hooks, and still emit tool.result with format_error: true. No behavior change intended.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Recover from malformed tool calls and stop after repeated format errors

1 participant