Skip to content

test(ai,core): meta-validate every built-in tool schema against the draft 2020-12 meta-schema after Anthropic translation — #1550's offender is still unnamed #1668

Closed

Description

@dezren39

Problem

Issue #1550 is CONFIRMED LIVE and its fix is blocked: route attempt 228433 (2026-11-04, workhorse → antigravity-direct/claude-sonnet-5-5) failed with the Anthropic-on-Vertex body

tools.5.custom.input_schema: JSON schema is invalid. It must match JSON Schema draft 2020-12

The task 9458 triage verdict (2026-11-04, on #1550) audited the in-repo translation layer and found it clean, and recorded the follow-up this issue files: nobody has ever meta-validated the actual built-in tool schemas — as translated for Anthropic targets — against the draft 2020-12 meta-schema. Until that test exists, the offender in tools.5 cannot be named: if a built-in schema is the invalid one, the test names it; if every built-in schema validates, the offender is proven external (MCP/plugin tool) and #1550's remaining step is only the route-ledger payload pull.

Evidence (origin/main 6a35d3a284, checked 2026-11-04)

  • No such test exists: a grep across packages/ for a 2020-12 meta-schema validation of tool schemas finds only schema generation sites — packages/core/src/tool/runtime.ts:169-170 emits schemas with target: "draft-2020-12" — and a fixture $schema string in packages/ai/test/provider/antigravity.test.ts:44. Nothing validates the emitted schemas against the meta-schema they claim to target.
  • The built-in tool set is enumerable: packages/core/src/tool/plugin/*.ts (read, edit, shell, grep, glob, patch, skill, question, subagent, …), each producing its inputSchema through definition() in packages/core/src/tool/runtime.ts; packages/core/test/tool-schema.test.ts already tests those definitions shape-by-shape but never meta-validates them.
  • Both Anthropic-bound translations are in scope: packages/ai/src/protocols/anthropic-messages.ts:550 passes input_schema: tool.inputSchema verbatim, and the Antigravity protocol's convertSchema (packages/ai/src/protocols/antigravity.ts, anthropic flag) re-serializes proto Schemas into Anthropic input_schemas for Claude leaves — the exact path that 400'd in Antigravity door: Claude leaf (claude-sonnet-5-5) 400s on workhorse attempts, Anthropic-shaped error #1550.
  • Oracle (per the repo's testing doctrine, AGENTS.md 2026-11-04): the official JSON Schema draft 2020-12 meta-schema itself — validation success/failure per named tool is decidable, no hand-written expectations.

Acceptance criteria

  • A test enumerates every built-in tool (the packages/core/src/tool/plugin/ set), takes each tool's generated inputSchema, applies the Anthropic-bound translation(s), and validates each result against the draft 2020-12 meta-schema.
  • A validation failure names the offending tool and the meta-schema violation (the test output identifies tools.N candidates instead of a bare pass/fail).
  • The test runs in an existing package test suite (core or ai) with no network access — the meta-schema is vendored or embedded as a fixture.
  • The result is recorded on Antigravity door: Claude leaf (claude-sonnet-5-5) 400s on workhorse attempts, Anthropic-shaped error #1550 either way: the named built-in offender, or the explicit "built-ins clean — offender is external" verdict that unblocks the payload-pull step.

Activity

  1. added
    kind/testTest coverage, test infrastructure, or test-only fixes
    task/triageQueue packet kind: verify, reproduce, and route this item
    source/creatorFiled by the new-issue-creator job
    priority/highImportant and time-sensitive
    queue/queuedHas a queued task in the Supabase agent queue
    app/aiPackage: ai (pkgs/ or apps/ in this repo)
    app/corePackage: core (pkgs/ or apps/ in this repo)
    effort/mMedium: expected under 1 day of work
    on Nov 4, 2026
  2. dezren39 commented on Nov 4, 2026

    @dezren39
    Author

    🤖 Agent note · filed

  3. added
    work/assignedAn agent holds this item via the queue (claimed task)
    queue/claimedA queue worker has claimed the task for this item
    and removed
    queue/queuedHas a queued task in the Supabase agent queue
    on Nov 4, 2026
  4. 13 remaining items

  5. dezren39 commented on Nov 5, 2026

    @dezren39
    Author

    🤖 Agent note · picked up

  6. added
    queue/parkedQueue task is parked (waiting on CI, time, or an event)
    status/parkedDeliberately paused with a recorded resume condition
    and removed
    queue/claimedA queue worker has claimed the task for this item
    on Nov 5, 2026
  7. dezren39 commented on Nov 5, 2026

    @dezren39
    Author

    🤖 Agent note · parked

  8. dezren39 commented on Nov 5, 2026

    @dezren39
    Author

    🤖 Agent note · state: adopted as design phase

  9. removed
    status/parkedDeliberately paused with a recorded resume condition
    queue/parkedQueue task is parked (waiting on CI, time, or an event)
    work/assignedAn agent holds this item via the queue (claimed task)
    on Nov 5, 2026
  10. dezren39 commented on Nov 5, 2026

    @dezren39
    Author

    🤖 Agent note · picked up

  11. dezren39 commented on Nov 5, 2026

    @dezren39
    Author

    🤖 Agent note · done

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    app/aiPackage: ai (pkgs/ or apps/ in this repo)app/corePackage: core (pkgs/ or apps/ in this repo)effort/mMedium: expected under 1 day of workkind/testTest coverage, test infrastructure, or test-only fixespriority/highImportant and time-sensitivesource/creatorFiled by the new-issue-creator jobtask/triageQueue packet kind: verify, reproduce, and route this item

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions