Skip to content

[syntax-error-quality] Error message quality: tools.github.toolsets type errors lack actionable guidance #29077

Closed

Description

@github-actions

📊 Error Message Quality Analysis

Date: 2026-04-29 | Tests: 2 | Average Score: 68/100 | Status: ⚠️ Needs Improvement

Summary: Engine type errors produce high-quality messages with valid values and examples. However, nested tools.github.toolsets type errors surface only a terse "got string, want array" message with no guidance on valid toolset names or correct array syntax — leaving developers without actionable help.

Analysis performed via static code review of pkg/parser/schema_compiler.go, pkg/parser/schema_errors.go, and the embedded main_workflow_schema.json since binary execution was unavailable in the sandbox.

Test Workflow Error Type Score Rating
1 research.md engine: 123 (wrong type) 82/100 ✅ Good
2 agent-performance-analyzer.md toolsets: "default" (string not array) 54/100 ❌ Poor

Strengths

  • Engine type errors are excellent: synthesizeOneOfTypeConflictMessage + knownOneOfFieldHints["/engine"] produce "expected string or object, got number. Valid engine names: claude, codex, copilot, gemini." with an example — actionable and complete.
  • Precise file:line:col location is attached to every schema error.

Weaknesses (top 3)

  1. No hints for /tools/github/toolsets — knownOneOfFieldHints in pkg/parser/schema_errors.go only has an entry for /engine. A type mismatch at /tools/github/toolsets produces only "got string, want array" with zero guidance on valid values.

    • Fix: Add to knownOneOfFieldHints: "/tools/github/toolsets": "Valid toolsets: all, default, action-friendly, repos, issues, pull_requests, actions, code_security.\n\nExample:\ntools:\n github:\n toolsets: [default]"
  2. No field example for toolsets — getFieldExample in pkg/workflow/schema_validation.go has examples for top-level fields but nothing for nested paths like toolsets. The generic array fallback only adds Example: toolsets: [item1, item2] which is unhelpful.

    • Fix: Expand fieldExamples to include "tools.github.toolsets" or use the schema enum values to auto-generate the hint.
  3. Engine example only shows string form — knownOneOfFieldHints["/engine"] shows engine: copilot but omits the object form engine: {id: copilot, model: ...}, leaving users discovering it via docs.

    • Fix: Add a second example line: # or with options:\nengine:\n id: copilot\n max-turns: 15

References:

Generated by Daily Syntax Error Quality Check · ● 6.7M · ◷

  • expires on May 2, 2026, 11:09 AM UTC

Activity

  1. pelikhan commented on Apr 29, 2026

    @pelikhan
    Collaborator

    @copilot add support for single string toolset field value

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions