Repository navigation
fix(session): recover from stalled model requests - #1426
Open
anandgupta42 wants to merge 11 commits into
Open
anandgupta42 wants to merge 11 commits into
anandgupta42 wants to merge 11 commits into
Conversation
A model request that never answered hung the session until an external kill, an idle-stream timeout ended it as an unknown error, and a stream that died part-way left its partial output in the message on retry. - Arm a 300s first-byte (response headers) timeout by default; it is configurable per provider through `options.headerTimeout`. - Watch Bedrock Converse binary event streams with the idle timeout too, not only SSE. - Classify a header timeout and an idle-stream timeout as retryable `APIError`s that say "The model stopped responding" (same mapping as upstream OpenCode), so the existing backoff retries them. - Before a retry, discard what the failed attempt streamed (text, reasoning, step-start, tool calls still pending). If the attempt already dispatched a tool call, do not retry, so the tool is never re-run; the session ends with an error naming the tool. - Never retry after a user cancel. Closes #1424 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- Refuse a retry once the failed attempt finished its step, so cost and tokens are not counted twice and the answer is not produced twice. - Explain on any `APIError` why a retry was refused, and stop logging that case as exhausted retries. - Keep a failed partial-output cleanup from escaping the catch block. - Do not append "gave up after N retries" when the user cancelled. - Test the discard rule directly. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configuration
📒 Files selected for processing (7)
Included review availability: This review used your included allowance. Your plan provides up to 4 included reviews per hour; 0 remain after this review. 📝 WalkthroughWalkthroughThe provider now applies header and stream timeouts, including to Bedrock event streams. Timeout errors can trigger bounded session retries. Before retrying, the session removes eligible partial output and prevents retry when a tool call has already been dispatched. ChangesStalled request recovery
Priority: ⬆️ High Estimated code review effort: 4 (Complex) | ~45 minutes Change: Bug fix · Severity of issue fixed: Medium Sequence Diagram(s)sequenceDiagram
participant SessionProcessor
participant ProviderFetch
participant MessageV2
participant StallRecovery
participant SessionRetry
SessionProcessor->>ProviderFetch: start stream attempt
ProviderFetch->>MessageV2: timeout or stream error
MessageV2-->>SessionProcessor: retryable APIError
SessionProcessor->>StallRecovery: discard partial attempt
StallRecovery-->>SessionProcessor: removed parts or dispatched action
SessionProcessor->>SessionRetry: wait before eligible retry
SessionRetry-->>SessionProcessor: backoff completed
Merge Risk: 🟡 Moderate · up to After a tool acts and the model stalls, the next turn can lack the tool’s history, allowing the model to repeat the action. Preserve that history before merging unless this limitation is explicitly accepted. 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 1📝 Generate docstrings 💡
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. I’m a rabbit watching headers arrive, Comment |
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: b5139e51-571b-463a-9747-dbdede1b5289) |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
All reported issues were addressed across 8 files
Reply with feedback, questions, or to request a fix.
View guided diff | Turn on auto-fix | Re-trigger cubic
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a0e99ddc5b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
- Use flat exports with a self re-export in `stall-recovery.ts`, per the package conventions. - Only the idle watchdog's `ResponseStreamError` is retryable; websocket transport errors keep their previous handling. - Make `SessionRetry.sleep` reject at once on an already-aborted signal so a Stop that lands just before the backoff is not delayed. - Log "max retry attempts reached" and append "gave up after N retries" only when attempts were actually exhausted. - Document that `headerTimeout: false` removes the hang protection. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: 71f9c537-4c70-4831-8b74-8145ece3058e) |
Replace the coercer reset on retry with `discardUnstarted()`, which drops only tool inputs that started streaming but were never called. Ids already allocated (including by earlier calls in the same message) stay reserved, so a retry that reuses a raw provider id can neither pair with the discarded start nor collide with an earlier call. Also make the stall tests deterministic: disable snapshot tracking in the test config (its background git work was what leaked an "All fibers interrupted" error at scope close) instead of sleeping. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: 7e582a30-82d2-4e20-a0a2-403013fd8df3) |
There was a problem hiding this comment.
All reported issues were addressed across 2 files (changes from recent commits).
Reply with feedback, questions, or to request a fix.
View guided diff | Turn on auto-fix | Re-trigger cubic
`discardUnstarted()` now removes the discarded attempt's allocations, so the per-id slot ordinals that `settled()` and `executionID()` index by stay aligned with what actually executes, while ids of earlier calls remain reserved. Tests cover both the collision and the ordinal alignment cases. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: f57ef6bb-8c5c-4794-80c4-8e832ac34505) |
![kilo-code-bot[bot]](https://avatars.githubusercontent.com/in/2193792?s=60&v=4){kind=small}
Code Review SummaryStatus: 2 Issues Found | Recommendation: Address before merge Overview
Issue Details (click to expand)CRITICAL
Files Reviewed (5 files)
Fix these issues in Kilo Cloud Review based on HEAD Previous Review Summaries (4 snapshots, latest commit f4a5cc7)Current summary above is authoritative. Previous snapshots are kept for context only. Previous review (commit f4a5cc7)Status: 4 Issues Found | Recommendation: Address before merge Overview
Issue Details (click to expand)CRITICAL
WARNING
Files Reviewed (12 files)
Fix these issues in Kilo Cloud Review based on HEAD Previous review (commit 2613ba5)Status: 3 Issues Found | Recommendation: Address before merge Overview
Issue Details (click to expand)CRITICAL
WARNING
Files Reviewed (3 files)
Fix these issues in Kilo Cloud Review based on HEAD Previous review (commit fc5f189)Status: 2 Issues Found | Recommendation: Address before merge Overview
Issue Details (click to expand)WARNING
Files Reviewed (7 files)
Fix these issues in Kilo Cloud Review based on current HEAD Previous review (commit 689b47e)Status: 8 Issues Found | Recommendation: Address before merge Overview
Issue Details (click to expand)CRITICAL
WARNING
SUGGESTION
Files Reviewed (10 files)
Fix these issues in Kilo Cloud Review based on current HEAD Reviewed by gpt-6-sol · Input: 92 · Output: 33K · Cached: 7.6M Review guidance: REVIEW.md from base branch |
- Replay an assistant step that ended with "The model stopped responding" and already ran tools, as an aborted step is, so the next turn knows those side effects happened instead of losing them from context. - Match the Bedrock event-stream content type case-insensitively. - Show "Model stopped responding" in the TUI for the retried-out error too. - Tighten the stall tests: assert the stalled request's own socket closed, start the backoff cancel only after the stall fired, clear test timers on every exit, and cover parts that predate the failed attempt. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: cdcb5346-922f-4c25-8e34-2372b02837e9) |
There was a problem hiding this comment.
All reported issues were addressed across 7 files (changes from recent commits).
Reply with feedback, questions, or to request a fix.
View guided diff | Turn on auto-fix | Re-trigger cubic
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: fc5f18953d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Narrow the replay exception for a stalled step to messages where a tool actually completed, so partial text or tool input that never ran is not fed to the next turn. Also wait for the torn-down requests' close events in the stall tests before asserting on them. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: 0987efd5-5531-46d2-add5-9c4cff0fee3c) |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2613ba519b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
The replay of an errored assistant step is a pre-existing limitation for every error, and handling it correctly (per-part filtering, all retryable errors, errored tools) is a separate change. Keep this PR to stall detection and the retry rules; the known limit is documented in the PR. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: 47e20d98-838e-40d0-a999-ba84bdf9f5db) |
…lean up atomically - The retry guard now also refuses once any tool's execute() began during the attempt, counted in the processor from the tool wrapper's `beginToolExecution`, because the AI SDK can start a tool before its part leaves `pending` or its tool-call event is persisted. - Remove a failed attempt's parts in one delete statement so a cleanup failure cannot leave the attempt half deleted. - Race the whole custom fetch against the first-byte deadline, so work such as Vertex credential acquisition that ignores the abort signal is bounded. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: 3e45f513-52a4-4e14-92ec-8fed25cfd73f) |
There was a problem hiding this comment.
2 issues found across 5 files (changes from recent commits).
Reply with feedback, questions, or to request a fix.
View guided diff | Turn on auto-fix | Re-trigger cubic
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="packages/opencode/test/session/stall-recovery.test.ts">
<violation number="1" location="packages/opencode/test/session/stall-recovery.test.ts:555">
P3: This callback is triggered by request arrival, not by creation of the pending tool part, so it can mark execution before any part exists. Wait for and assert the `call_1` part is pending before invoking `beginToolExecution`; the final `parts.some(...)` only checks state after the attempt.</violation>
</file>
<file name="packages/opencode/src/provider/provider.ts">
<violation number="1" location="packages/opencode/src/provider/provider.ts:2049">
P2: This race only observes the header deadline, so caller cancellation cannot settle the wrapper while a custom fetch is awaiting work that ignores `init.signal`. Vertex awaits credential acquisition before forwarding the signal, leaving the request pending until credentials resolve or the 300-second timeout; race the request signal as well.</violation>
</file>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b852ba47d6
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Race the custom fetch against the combined request signal (first-byte deadline, caller cancel, timeouts), not only the deadline, so a Stop is not held up by pre-request work that ignores the signal. The tool-race test now waits for the pending tool part before reporting the execution started. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: f4e7bfbf-6fb1-4f17-a074-5a8c4ea5300d) |
| const removed = new Set<string>(stale.map((part) => part.id)) | ||
| if (stale.length > 0) { | ||
| Database.use((db) => { | ||
| db.delete(PartTable) |
There was a problem hiding this comment.
CRITICAL: Persist retry cleanup in the synchronized session event log
This raw PartTable delete removes the failed attempt locally, but the following Bus.publish(MessageV2.Event.PartRemoved, ...) is only a legacy notification; it does not append SessionV1.Event.PartRemoved to the durable core event log. The shipped workspace sync replays that log into another workspace via SessionProjector, which has the original PartUpdated events but no removal. After a stalled attempt is retried, the other workspace therefore retains the discarded text, step-start, or pending tool alongside the retry's output, and future sync cannot correct it. Publish/project durable part-removal events as part of the atomic cleanup, rather than deleting only the local projection.
Reply with @kilocode-bot fix it to have Kilo Code address this issue.
Contributor
Author
There was a problem hiding this comment.
Not changed in this PR. The cleanup persists a removal the same way the existing Session.removePart in session/index.ts does: a delete on PartTable followed by Bus.publish(MessageV2.Event.PartRemoved, ...). Session.updatePart writes parts through the same path. The bulk delete replaced a loop over Session.removePart so the cleanup is one statement; it did not change which event log the removal reaches. If removals made through this path are not replayed by workspace sync, that applies to every existing caller of Session.removePart and is a separate change to how this module persists part events, outside the stall-retry fix.
anandgupta42
added a commit
that referenced
this pull request
Oct 12, 2026
A `ProviderError.HeaderTimeoutError` (the first-byte deadline, 10 s on the
OpenAI path) fell through `MessageV2.fromError` to `UnknownError`, which the
session never retries — one slow provider accept ended an otherwise healthy
run with `UnknownError: Provider response headers timed out after 10000ms`.
Classify it as a retryable `APIError` ("The model stopped responding: no
response headers within Ns"). Nothing was streamed for that attempt, so
re-issuing the request is safe; other `ResponseStreamError`s keep their
non-retryable handling. Wording and constants match PR #1426 so that branch
merges cleanly on top.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
anandgupta42
added a commit
that referenced
this pull request
Oct 12, 2026
…requests
Traces from 270 Opus 5.5 evaluation runs showed:
- every `sql_execute` against a MongoDB warehouse was denied as a write
(52 of 52): the SQL classifier reads a JSON command document as a write.
Mongo warehouses now classify by command (`find`/`aggregate` without
`$out`/`$merge`/`countDocuments`/`distinct` are reads; drops are blocked),
and the native `{"find": "coll", …}` wire shape is accepted everywhere
(`normalizeMqlCommand` in the driver, used by `sql_execute` and
`sql_materialize`, which rejected it as command "undefined").
- one evaluation run stalled 890 s on a single Bedrock request: providers without a
header timeout now default to 300 s, and the idle-stream watchdog also
covers Bedrock's `application/vnd.amazon.eventstream` bodies (both as in
PR #1426).
- analyst prompt: every clause must bite; compute both readings of an
ambiguous phrase and choose by the question's own definitions.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This was referenced Oct 14, 2026
anandgupta42
added a commit
that referenced
this pull request
Oct 16, 2026
* feat(analyst): verifiable data analysis — deadlines, result envelopes, scratch bridge, checks, skills
Analyst mode becomes a general-purpose, verifiable data-analysis agent. Every
change is engine-agnostic and applies beyond individual evaluation runs; the
evaluation traces document the development history and evidence behind each item.
Query execution
- Default 120 s execution deadline on every warehouse (`queryTimeoutMs` per
call > connection `query_timeout_ms` > env `ALTIMATE_SQL_QUERY_TIMEOUT_MS`;
`0` disables). Postgres uses `statement_timeout` (never loosening a stricter
server default), Mongo `maxTimeMS`.
- SQLite and DuckDB run in one supervised subprocess per connection: Bun
`Worker.terminate()` does not preempt a blocking native call and the legacy
`duckdb` binding's `interrupt()` is a no-op, so cancellation kills and
respawns the helper (lock released, state loss disclosed). The helper starts
in a private 0700 directory with a minimal environment and an empty
`bunfig.toml`, so project preloads and credentials never reach it.
- `sql_execute` leads row-limited or clipped results with a `Preview: N of M
returned rows … saved to <path>` envelope, caps the body at 50 rows / 16 KiB,
spills all returned rows as typed JSONL, validates `limit` (1–10,000), and
exposes `query_timeout_ms`.
- Driver and provider errors are surfaced: a stream that ends with finish
`other`, no content and no usage is reported as a decode failure instead of a
silent empty answer (Bedrock event-stream decoding under Bun; pins
`@smithy/util-buffer-from`/`is-array-buffer` to 4.2.2, see also #1425).
Cross-engine analysis
- Session-scoped in-memory `scratch` DuckDB plus `sql_materialize`: run a
read-only query on any connected warehouse (SQL engines and Mongo reads) and
store all rows as a typed scratch table, so joins across engines happen in
the engine rather than by copying key lists through the conversation.
- `sql_check`: executable result invariants — fan-out, join-key profile (exact
vs normalized matches, format differences), grain uniqueness, null share,
extraction-regex coverage — with a verdict line and numeric metadata.
- `schema_inspect` appends value samples (distinct values for low-cardinality
columns, quoted examples for text columns so whitespace/case variants show,
null share); empty or near-empty results with a literal filter get a probe
hint.
Agent surface
- Analyst prompt rewritten as a short analytical method (grain, population,
denominator; inspect rows before joining; derive in-engine; treat inferred
labels as data) ending in a delimited `**Answer:**` line that may honestly be
"unable to determine". The mandatory validate/analyze ritual is gone (an A/B
showed no demonstrated score benefit).
- Analyst can load skills and delegate to a new read-only `data-explorer`
subagent (its only allowed `task` target; bash fully denied there). New
built-in `data-question` skill with synthetic worked examples. Embedded
built-in skills are no longer hidden by one old installed skill.
- Reusing a `task_id` from a foreign session now starts a fresh child instead
of adopting that session's permissions.
`run` lifecycle and traces
- `--max-turns` exhaustion gets exactly one best-effort finalization
generation (exit code and reason unchanged); `--format json --output` writes
the completed answer instead of a placeholder.
- Traces cover descendant (subagent) sessions, reasoning text when the provider
exposes it, and the effective system prompt and resolved tool schemas per
generation (hash-deduplicated); resume no longer overwrites earlier child
evidence. Truncation hints name a subagent the current agent can call.
Tests: drivers 416, opencode suite green except `mcp.headers` (fails
identically on `main`); live end-to-end analyst smoke suite
(`scripts/analyst-smoke.sh`) 11/11.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* test: restore the genuine duckdb driver after module mocks
Four test files mock `@altimateai/drivers/duckdb` at module load. Bun's
`mock.module` leaks across files in the shared test process, so every
DuckDB-backed test that ran later (scratch, sql_materialize, sql_check,
schema_inspect value samples) saw a fake engine returning empty rows and
failed only in full-suite runs. Each mocking file now snapshots the genuine
exports before mocking and restores them in `afterAll`.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* fix: DuckDB loads with `--dir` in argv, metadata statements are reads, provider list retries at startup
Three defects surfaced by the driver probe during evaluation runs and by load testing:
- The `duckdb` package locates its addon through `@mapbox/node-pre-gyp`, which
parses the HOST process argv and abbreviation-matches an unrelated `--dir`
flag to its `--directory` option, joining that value onto the absolute
manifest path. `altimate-code run --dir <x>` therefore failed every DuckDB
load with ENOENT under `<x>/usr/lib/...` (the 2026-08-31 root cause; never
merged). The driver now only resolves the module path in the parent and lets
the helper process, which has a clean argv, load the binding. Real-process
regression test with `--dir` in argv.
- `SHOW`, `DESCRIBE`, read-only `PRAGMA` and `EXPLAIN` of a read were classified
as writes ("other" category), so read-only agents were hard-denied from
listing tables on DuckDB/SQLite. They now classify as reads; `EXPLAIN ANALYZE`
of a write, `PRAGMA x = v`, `SET`, `USE` stay writes.
- `run` pre-validates an explicit `--model` against `sdk.provider.list()`; a
failed list call under startup load was read as an empty catalogue and
reported as "Model not found" (the cold-start race that cost evaluation runs in
August). It now retries three times and surfaces the real error.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* test: metadata statements classify as reads; SET/USE still prompt
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* fix(session): retry a request whose response headers never arrived
A `ProviderError.HeaderTimeoutError` (the first-byte deadline, 10 s on the
OpenAI path) fell through `MessageV2.fromError` to `UnknownError`, which the
session never retries — one slow provider accept ended an otherwise healthy
run with `UnknownError: Provider response headers timed out after 10000ms`.
Classify it as a retryable `APIError` ("The model stopped responding: no
response headers within Ns"). Nothing was streamed for that attempt, so
re-issuing the request is safe; other `ResponseStreamError`s keep their
non-retryable handling. Wording and constants match PR #1426 so that branch
merges cleanly on top.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* fix(analyst): harness defects found in the first evaluation traces
Trace audit of 51 evaluation runs (Opus 5.5 and gpt-5.6-sol, analyst agent)
surfaced general defects in the data-agent harness; none is specific to an
evaluation run.
- `sql_check`: strict-schema callers (OpenAI Responses API) send every
property and fill unused ones with "" / []; the strict zod schema rejected
49 of 56 calls. The published schema now tolerates empty values, which
`dropEmpty()` removes before the strict parse.
- `sql_materialize`: the row cap was not a memory bound — a >2M-row Postgres
result was fully buffered before the cap rejected it and the kernel
OOM-killed the process (8.3 GB RSS). Pre-flight `count(*)` + 1,000-row
sample now estimates size and refuses before extraction; `max_bytes`
(default 256 MiB, hard 2 GiB) joins `max_rows`; 0 means default.
- `sql_execute`: DDL/DML on the in-memory `scratch` database no longer asks
for `sql_execute_write` (read-only agents denied 8 `CREATE TABLE` calls
that `sql_materialize` was allowed to do anyway); the hard-deny list stays.
- `schema_inspect`: a blank `schema_name` means unset, and an empty column
list is reported as "table not found" instead of a valid empty schema
(21 of 44 Postgres inspections read as success).
- `run --output`: the file now holds the text of the last model step only;
commentary from earlier steps ("Let me check all formats.") leaked into 9
of 31 answers. Everything is kept as the fallback when no reply was given.
- `bin/altimate-code`: a child killed by a signal exited 0; now 128+signal
with a stderr line, so an OOM-killed run is not recorded as success.
- analyst prompt: the final message is the deliverable — no excluded
candidates or alternative answers, exact format when one is requested,
N concrete rows for a top-N with a boundary tie.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* feat(analyst): surface silent-wrong lint findings with query results
Traces show the model computing wrong numbers silently (integer division,
averages of averages, fan-out after joins, arbitrary LIMIT) and never calling
the separate lint tools. `sql_execute` and `sql_materialize` now run the
altimate-core linter on the executed SQL (warehouse dialect, fresh cached
schema when available) and append up to three `Check:` lines from a curated
allowlist of correctness rules, Warning severity and above. Advisory only:
errors are swallowed, a lint over 250 ms disables further lints for the
session, `ALTIMATE_SQL_EXECUTE_LINT=0` turns it off. Fired rules are recorded
in the tool metadata (`lint_rules`).
The allowlist already names `integer_division`, `limit_without_order_by`,
`avg_of_avg` and `reserved_word_alias`, which ship in the next altimate-core
release.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* fix(schema): resolve the engine default schema in schema.inspect
`schema.inspect` defaulted `schema_name` to "public", which is Postgres's default
but not DuckDB's or SQLite's ("main"): every DuckDB inspection without an
explicit schema returned no columns, and an evaluation run read that as
"table not found" while the table was queryable. Without a schema the handler
now starts from the engine default and then searches the listed schemas for
the first that has the table; an explicit schema is used as given.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* fix(analyst): address review findings on the v2 harness changes
- `sql_execute`: the scratch exemption covered every write, including
`COPY … TO`, `ATTACH`, `INSTALL`/`LOAD`, `EXPORT` and `SET`, which reach
outside the in-memory engine. Only table/view DDL and row DML
(`isScratchSafeWrite`) skip the write permission now.
- `run --output`: the completion check and the post-compaction nudge ask for a
bare `DONE`; that acknowledgement replaced the saved answer. The collector
keeps the reply it confirms; substantive text still replaces it.
- `sql_check`: a coverage check's `pattern: ""` is a value (match empty
strings), not an omitted field.
- header-timeout `APIError` message says "timed out" so run accounting keeps
attributing an exhausted retry to a timeout.
- `json-output` integration test expects the final-step-only `--output`.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* fix(analyst): bound schema probing, materialize pre-flight and lint schema size
Review follow-ups on the v2 harness changes:
- `schema.inspect`: derive the engine from the connection actually used when
`warehouse` is omitted; try the engine default schema first and list schemas
only on a miss, skipping system schemas and probing at most 20; value
profiling targets the schema the handler resolved.
- `sql_materialize` pre-flight: count through a `LIMIT max_rows + 1` derived
table so an oversized source is rejected after bounded work, and skip the
1,000-row sample once the cap is known to be exceeded.
- `sql-result-lint`: build the lint schema only from the tables the SQL
references (via `extractMetadata`), capped at 2,000 columns; the synchronous
lint is now ~20 ms on a 500,000-column cache instead of blocking the loop.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* fix(analyst): pre-flight MongoDB sources before materializing them
Mongo sources skipped the materialize pre-flight, so `find`/`aggregate` results
were buffered up to the row cap before any check: an evaluation run on the
articles collection was OOM-killed at 26 GB RSS. `find` now counts with `countDocuments` bounded by
`limit: max_rows + 1`; `aggregate` appends `$limit`/`$count`; both refuse
before extraction when over the row cap and otherwise sample 1,000 documents
for the byte estimate, exactly like the SQL path. The driver passes
`limit`/`skip` through to `countDocuments`.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* fix(analyst): read MongoDB commands as reads, bound stalled provider requests
Traces from 270 Opus 5.5 evaluation runs showed:
- every `sql_execute` against a MongoDB warehouse was denied as a write
(52 of 52): the SQL classifier reads a JSON command document as a write.
Mongo warehouses now classify by command (`find`/`aggregate` without
`$out`/`$merge`/`countDocuments`/`distinct` are reads; drops are blocked),
and the native `{"find": "coll", …}` wire shape is accepted everywhere
(`normalizeMqlCommand` in the driver, used by `sql_execute` and
`sql_materialize`, which rejected it as command "undefined").
- one evaluation run stalled 890 s on a single Bedrock request: providers without a
header timeout now default to 300 s, and the idle-stream watchdog also
covers Bedrock's `application/vnd.amazon.eventstream` bodies (both as in
PR #1426).
- analyst prompt: every clause must bite; compute both readings of an
ambiguous phrase and choose by the question's own definitions.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* fix(analyst): durable scratch store and bulk-loaded materialize
Three of five sol trials on one question timed out: materializing a 3.3M-row
table into scratch took 116–552 s through 500-row VALUES batches, and when a
later `sql_check` hit the 120 s deadline the local-engine restart reopened the
`:memory:` scratch database empty — every scratch table vanished and the trial
redid the work.
- scratch is now a private per-process DuckDB file (0700 temp directory,
removed on close/exit; the path is redacted from errors), so a helper
restart after a timeout reopens the same catalog.
- `sql_materialize` writes extracted rows to a temp JSONL file and loads it
with `read_json(... columns={...})` using the inferred types: 1,000,000 rows
from SQLite 11.2 s → 2.3 s (bench script included).
- analyst prompt: move the smaller side of a cross-engine join — a few hundred
keys belong in an IN list, not a multi-million-row materialization.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* feat(run): wall-clock time budget with a soft nudge and bounded finalization
`run` had a turn budget but no elapsed-time budget, so a session under an
external kill (evaluation runs invoked as `timeout 900 altimate-code run …`)
could explore until it was killed with no answer — one spawned 11 labeller
subagents for 905 s with a defensible partial answer minutes earlier.
- `--time-budget <seconds>` (env `ALTIMATE_RUN_TIME_BUDGET_S`), validated
like `--max-turns`.
- at 80 % of the budget one continuation directive asks the model to stop
exploring and give its best current answer (`REPLY_BEFORE_TIME_BUDGET`),
delivered at the next step boundary, once.
- at 100 % the session is aborted like the turn budget and gets the same
bounded finalization so `--output` holds whatever answer it can give;
exit code and JSON events mirror the turn-budget semantics. Subagent time
counts.
- `time_budget_s`, `elapsed_s` and the deadline that fired are recorded in
run accounting and the trace metadata.
- `--output` now writes only the harness note when a run ends without a
reply (earlier commentary is not an answer).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* fix(session): keep BigInt tool values out of request serialization
An evaluation run died with `UnknownError: JSON.stringify cannot serialize BigInt.`
on the model request following a DuckDB `COUNT(DISTINCT …)` tool result: the
output-token estimator stringifies the outgoing messages before `llm.request`
is published, and a native BigInt in a tool value reaches it unconverted.
- `altimate/util/json.ts`: `bigintReplacer` and `normalizeBigInts` (copies
JSON-shaped data, preserving SDK media objects and buffers).
- `session/llm.ts`: normalize the outgoing messages before estimation and SDK
serialization; the stored tool result is untouched.
- tracing, run tracing and the TUI worker serialize with the BigInt replacer;
a tracing failure can no longer surface as a session error.
- regression tests at the processor boundary, tracing, and the sql_execute /
sql_materialize tool results with BigInt values and a lint finding.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* chore(markers): wrap run output and budget code in altimate_change markers
The strict marker check passes locally only because the local `upstream/dev`
ref now points at the restructured opencode v2 tree, where none of the shared
files exist at their v1 paths. Checked against the pinned `v1.18.34` tree (and
with no upstream remote, as CI falls back to), `run.ts`, `run-accounting.ts`
and `termination.ts` had unmarked additions. Comment-only change.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* fix(tools): apply read-permission rules to grep, glob and ls results
A read-denied directory could still be searched: `grep`, `glob` and `ls`
returned matching lines and paths from files the `read` rules deny (seen in an
evaluation run, where a search surfaced another run's evaluation traces).
Search results now pass through the same permission evaluation the read tool
uses: denied paths are dropped silently, "ask" paths are batched into one
prompt covering only the returned rows, and the 100-row cap applies after
filtering (grep stops at 101 permitted matches or 10,000 scanned). Tool
contexts carry the session's permission ruleset; `batch` forwards it.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* fix(schema_inspect): resolve a warehouse name given as the schema
`sql_materialize` advertises results as `scratch.<table>`, so models call
`schema_inspect` with `schema_name: "scratch"` (14 failures in one evaluation
run) and get "cannot find a queryable table". When `schema_name` or the
leading qualifier of `table` names a configured warehouse and no such schema
exists in the current one, the lookup resolves against that warehouse's
default schema and says so; an explicit `warehouse` argument still wins.
Not-found errors now list the warehouses and the addressing form.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* fix(provider): retry a model stream that goes silent, for every provider
Evaluation runs lost complete work when a streaming response went idle for
400-700 s after the last tool call; the run's time budget then fired with no
answer to salvage. The chunk watchdog that only covered Bedrock's eventstream
now wraps every streaming body (`text/event-stream` included) and aborts a
silent stream with a retryable `MODEL_STOPPED_RESPONDING` error, so the
session's bounded retry re-issues the request. Default stays 300 s
(`chunkTimeout`, PR #844); `ALTIMATE_STREAM_IDLE_TIMEOUT_MS` overrides it and
`0` disables.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* fix(sql): close the literal-masking bypass and gate reads on parsed statements
Review of PR #1441 found that the read-only SQL tools could run hidden
writes: the masker treated `\\` as a complete string, so a PostgreSQL
`E'\'--'; DELETE …` text hid the second statement from both the masker and
the classifier, and `postgres.ts` runs such text as one simple-protocol
query. Quoted `--` or `/*` likewise hid CALL, SET ROLE, ATTACH and friends,
and argument-form PRAGMAs that write (`user_version(5)`, `journal_mode(...)`)
were classified as reads.
- `sql-tokenize.ts`: a string-literal-aware tokenizer (`''`, `E'…'`
escapes, dollar quoting, quoted and bracketed identifiers, nested block
comments; unterminated input is incomplete).
- `sql_check`, `sql_materialize` and the result lint accept only a single
parser-reported query; `sql_execute` accepts statements the classifier
proves read (SHOW, DESCRIBE, EXPLAIN of a read, allowlisted PRAGMAs) and
refuses multi-statement reads, which the connectors cannot return.
- Comment stripping and `otherIsReadOnly` evaluate per parsed statement; a
parser failure is not a read. Writing PRAGMAs need write permission;
bare pragmas and a small read allowlist stay reads.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* fix(provider): retry idle streams only before output, keep header timeout opt-in
The idle-stream retry re-issued a request after text or tool calls had
already been emitted, so a tool could execute twice; the retry now applies
only when nothing was emitted in that attempt. The 300 s header deadline is
opt-in again for non-OpenAI providers instead of applying to every request
including non-streaming ones. Run traces cap user and synthetic attachment
text at the same 4000 characters as logged messages and serialize API errors
without response headers or unbounded bodies. The help snapshot gains the
`--time-budget` option that CI compared against.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* fix(scratch): isolate scratch stores per root session; search rules only tighten
The scratch connector was one process-global store, so sessions served by
one process could read, overwrite or drop each other's tables, and the
scratch write exemption let that happen without a prompt. Stores are now
keyed by the root session id (subagents share their parent's tables, as the
labelling workflows rely on) and closed when the session is removed. The
search-result permission filter evaluates the worktree-relative path exactly
as the read tool does and uses absolute rules only to deny or ask, never to
allow.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* fix(drivers): run the local-engine helper in the caller's cwd with a usable env
The DuckDB/SQLite helper was spawned in its private temp directory with a
stripped environment, so relative paths in SQL (`read_csv_auto('data.csv')`,
`ATTACH 'other.db'`) resolved against the wrong directory, relative writes
vanished on cleanup, TZ and installed extensions were lost, and `close()`
queued behind a running query for up to the query deadline. The helper now
runs in the caller's working directory (temp dir kept for HOME, XDG, TMP and
DuckDB spill), passes TZ, proxy and documented credential variables, points
DuckDB at the real extension directory, and `close()` bypasses the queue
(2 s polite grace, 500 ms when a query is in flight). Rows decode as plain
objects and BLOBs as Buffers so STRUCT values format. Mongo native
`count`/`distinct` commands forward their `query` filter.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* feat(run): interrupt the in-flight generation at the soft time budget and lower reasoning effort
An evaluation run showed the failure this fixes: the model started a generation
at +85 s, produced its last visible reasoning chunk at +98 s, then reasoned at
effort "high" with the stream alive until the hard abort at 840 s; no answer
was ever written. The soft deadline could only queue a nudge behind the
in-flight request.
- At the soft deadline the run aborts the in-flight model REQUEST only (per
attempt controller combined with the session signal), after any executing
tool has persisted its result; emitted text and reasoning parts are kept,
the attempt is not retried as a provider error, and the "answer now" nudge
becomes the next turn (`x-altimate-run-budget` header on the prompt route).
- For the rest of that run, including the post-hard-abort finalization, every
model request is sent with the provider's lowest reasoning effort
(OpenAI-style `reasoning_effort`, Anthropic/Bedrock thinking disabled,
Gemini/Mistral equivalents) without touching the configured variant; task
subagents inherit it. `ALTIMATE_RUN_BUDGET_EFFORT_DOWNGRADE=0` disables the
downgrade (the interrupt still happens).
- `run` exits 0 when the finalization wrote an answer and reports the budget
abort as a warning; Ctrl-C, hard failures and silent finalizations stay
non-zero.
- Trace metadata gains `generation_interrupted_at` and `effort_downgraded_at`
(Unix ms) next to the existing `time_budget_*` fields.
Runs without `--time-budget` are unaffected.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* fix: close review findings on permission ceilings, read-only extraction and stream handling
Review-bot findings of the highest two severity tiers, each with a regression
test:
- `data-explorer` keeps its read-only ceiling after agent or global permission
overrides; resumed tasks are bound to the agent that created them.
- PostgreSQL extraction runs in a pinned READ ONLY transaction and rolls back;
Mongo aggregations refuse `$out`/`$merge` anywhere in the pipeline
(including `$unionWith`, `$lookup` and `$facet` sub-pipelines); attached
catalog targets need write approval; unbounded `distinct` is refused.
- Request snapshots travel on a private bus channel exposed only through a
guarded trace subscription; exported errors are redacted before truncation.
- The hard deadline aborts queued requests, custom fetch hooks race the abort
signal, idle-stream replay safety is tracked at decoded output, and the
dash-comment lexer is dialect-aware (MySQL `a--b` is arithmetic).
- Scratch stores stay restricted after a helper reopen and are closed on
registry shutdown; the helper keeps default AWS profile paths; stdio readers
scan chunks once; schema inspection uses the configured default schema and
routes warehouse aliases before probing; value profiling honours execute
permission and workspace routing.
Lower-severity findings are triaged in the PR and left for follow-up.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* fix: bound Mongo aggregate materialization and always sanitize resumed reasoning
Two review findings of the highest severity tiers, each with a regression
test:
- `sql_materialize` appends a terminal `$limit` to Mongo aggregate
pipelines so an expanding stage after an earlier `$limit` (such as
`$unwind`) cannot exceed the row cap.
- Interrupted-reasoning message sanitization now runs for every request,
not only when a run budget is active, so a resumed Anthropic turn never
carries an empty assistant block.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* fix: recognise dollar-quoted strings only on dialects that support them
A review finding of the highest severity tier, with regression tests.
`$tag$…$tag$` was treated as a string literal on every dialect, and some
statement-type checks ran without a dialect, so on engines where `$` is an
identifier character a write after a semicolon inside that span was
classified as a single read and skipped the write prompt.
- The tokenizer recognises dollar quoting only for `postgres`, `redshift`,
`duckdb` and `snowflake`; for every other, unknown or omitted dialect the
span is ordinary text, so `SELECT $x$; DELETE FROM t; $x$` is three
statements including a write and is refused under read-only permission.
- Every statement-type, classification, fingerprint and lint call on the
execute, materialize and check paths now receives the connection's
canonical dialect.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* fix(drivers): share one DuckDB engine per store across concurrent connectors
Each DuckDB connector spawned its own local-engine helper process, so
several connectors opening the same store at once became several OS
processes competing for DuckDB's exclusive file lock. The "many connectors
at once" E2E failed in CI with "Conflicting lock is held in . (PID …)" while
passing locally, where the opens happened to serialize.
- A process-local registry keyed by resolved path, access mode, sandbox
directory, working directory and extension directory shares one helper
per store, reference-counted; the last connector to close shuts the helper
down, and a replacement open waits for a retiring helper to finish.
- Inside the shared helper each connector gets its own native connection,
preserving transaction and temporary-table isolation; `disconnect`
releases one session.
- READ_ONLY connectors use a separate read-only engine; a foreign lock held
by another process still surfaces as the existing lock error.
- Connect and close are idempotent, including calls racing their own
pending open or close.
Tests: 14 deterministic unit tests for the shared engine (barrier-held
opens, ownership, shutdown ordering, recovery, configuration isolation) and
three real-store E2E regressions (forced overlap with final lock release,
independent sessions, overlapping read-only connectors). The warehouse E2E
now closes its connector before resetting the registry; the shared engine
exposed that leak.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* fix: authorize and validate SQL before routing, block executable comments, bound extraction bytes
Five review findings of the highest severity tiers, each with a regression
test written red first:
- `sql_check` validates every supplied SQL field as read-only and asks for
`sql_execute` permission before any workspace redirect or connection, so a
caller cannot follow a redirect to an integration tool with write SQL or
without SQL-read authorization.
- Executable comments (`/*! … */`, `/*M! … */`) are rejected before
permission checks and dispatch; generic parsers discard their bodies, so a
hard-denied statement inside one previously passed classification.
- The DuckDB helper forwards the documented Azure storage authentication
variables and nothing else from the parent environment.
- Source extraction enforces `max_bytes` while rows are being collected for
SQLite, DuckDB, PostgreSQL and MongoDB (a preflight sample no longer bounds
wider later rows); drivers without bounded transfer are refused rather than
buffered unbounded. This is a serialized-row ceiling, not a heap limit.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* test(drivers): prove sandbox restrictions hold on shared DuckDB connector sessions
A review finding suggested that per-connector native connections inside the
shared helper could miss the sandbox settings. They cannot: every restriction
the helper applies (`allowed_directories`, `enable_external_access`,
`autoinstall_known_extensions`, `autoload_known_extensions`,
`lock_configuration`) is a GLOBAL DuckDB setting in the pinned 1.4.4 binding,
and the configuration lock applied last rejects any later change on any
connection. HOME isolation is process-level, set before the database exists.
- Document the global scopes and why the restrictions are not replayed per
connection in the helper source.
- Add a real-binding E2E (`keeps sandbox restrictions on shared connector
sessions`, normal and streamed results) holding two writer connectors on one
sandboxed store: each session's first statement is an outside-file or HTTPS
read and is denied with the configuration-denial error; positive controls
read inside the allowed directory and verify independent temporary tables.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* fix: gate materialization on SQL-read permission, strip tools during budget finalization, clean helper spills
Three review findings of the highest severity tier, each with a red-first
regression test:
- `sql_materialize` asks for `sql_execute` permission before opening any
source, so a ruleset that denies SQL reads cannot extract through
materialization.
- Budget finalization removes executable tools and historical tool stubs
from the finalization request and restores them afterwards, so a model
that emits a tool call during finalization cannot run it; exactly one
`budget_exhausted` event is emitted per run even when finalization takes
an extra step.
- Each DuckDB helper's spill directory is deleted after its confirmed exit
(including timed-out helpers); live helpers keep their spills.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* fix: keep the caller's DuckDB secret directory and resolve symlinks in read-permission checks
Two review findings of the highest severity tier, each with a red-first
regression test:
- Shared DuckDB helpers set the caller's persistent secret directory after
open, so previously stored secrets load and new ones persist across close
and timeout; scratch stores stay isolated.
- `glob` and `ls` absolute read-permission rules check the canonical target
of directory symlinks as well as the lexical path, so a symlink into a
denied directory cannot bypass the rule; relative and lexical restrictions
are preserved (`grep` already canonicalised its root).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* fix: ask SQL-read permission before materialize routing; tolerate unresolvable search candidates
Review findings, each with a red-first regression test:
- `sql_materialize` requests `sql_execute` permission for the supplied SQL
and source before the workspace-precedence redirect, so a shadowed
warehouse cannot route a read past a configured denial; an allowed call
still redirects as before.
- The canonical-path read-permission check in `glob`, `ls` and `grep`
handles resolution failures per candidate: a path that cannot be
canonicalised (`EACCES`, a vanished file, a symlink loop) is omitted and
the rest of the results are returned instead of the whole search failing.
- The directory-permission test skips with a message when run as root,
where `0o400` does not restrict traversal.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* fix(drivers): apply the resolved Mongo deadline to write operations
A review finding of the highest severity tier: the resolved `maxTimeMS`
reached Mongo reads but not the nine write paths (CRUD, collection and
index operations), so a write could run unbounded. Every write now forwards
the deadline while preserving its command options; override, disabled and
error cases are covered.
Test: `forwards the default maxTimeMS and preserves command options` across
the nine write variants.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* test: close the lock holder's connector on SIGTERM; match the dialect-aware fingerprint call
- `duckdb-lock-helper.ts`: the foreign write lock is held by the driver's helper
process, a child of the holder script, not by the script itself. A bare SIGTERM
orphaned that helper with the lock still taken until it noticed its parent was
gone, so the next open in the suite collided with it on slow runners
("locked by another process" in `keeps overlapping connectors in independent
sessions` and the read-only sharing test). The script now closes its connector
on SIGTERM, which kills and reaps the helper before exiting; `release()` falls
back to SIGKILL after 10 s.
- `telemetry-signals.test.ts`: the source-order assertion matched
`computeSqlFingerprint(args.query)` literally; the call now passes the resolved
dialect, so match the query-argument prefix instead.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* fix(run): keep adaptive thinking at the lowest effort when the time budget lowers reasoning
Always-thinking Claude models (Opus 4.7 and later, Fable 5) reject `thinking.type: "disabled"` with HTTP 400 ("use thinking.type.adaptive and output_config.effort"), so every budget-exhausted final request failed and the best-effort reply was lost.
- `applyBudgetEffort` now returns the provider's lowest adaptive variant for those models: direct Anthropic/Vertex/Gateway get `thinking: adaptive` plus `effort: low`; Bedrock gets `reasoningConfig: adaptive/low` and matching raw `thinking`/`output_config.effort` fields, dropping any stale token budget
- Claude 4.6 and older keep the disabled-thinking path
- `ProviderTransform.anthropicOmitsThinking` is exported as the detector
- tests: adaptive-only models on direct Anthropic and Bedrock keep adaptive/low and never emit `disabled`; the 4.6 expectations are unchanged
Live check on Bedrock Opus 5.5 with an 8 second budget: before, the finalizer request failed with the 400 and the run ended in error; after, the lowered-effort request succeeded and the run ended with a final reply.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* fix(run): keep adaptive thinking for always-thinking Claude models on the SAP provider during budget finalization
The SAP `modelParams` branch of `applyBudgetEffort` still forced `thinking.type: "disabled"` for every Anthropic id, so Opus 4.7+ and Fable 5 served through `@jerome-benoit/sap-ai-provider-v2` got the same HTTP 400 on the budget-exhausted final request. The branch now keeps the lowest adaptive variant for those models and only disables thinking for Claude 4.6 and older. Tests: SAP Opus 4.7 keeps adaptive/low with unrelated model parameters intact; SAP Sonnet 4.6 still disables thinking.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* fix(analyst): validate source writes before routing, keep BSON binaries lossless, and make absolute read restrictions authoritative in search tools
- `sql_materialize`: `assertReadOnlySource` runs before the `sql_execute` ask and before the workspace redirect, so a write statement or Mongo write pipeline against a shadowed warehouse is rejected instead of redirected to the integration tool
- Mongo driver: BSON `Binary` normalises to `base64:<bytes>` (subtype 4 to its canonical UUID string when `toUUID()` exists); schema inference reports `uuid`/`binary`; distinct same-length binaries no longer collide in scratch tables
- search read approvals (`grep`, `glob`, `ls`): the final `read` ask carries a ruleset that re-applies absolute read rules after the relative/wildcard fallbacks, so `"/project/private/**": "ask"` prompts even with a trailing `"*": "allow"`; `Tool.Context.ask` accepts an optional `ruleset`, honoured by the prompt, session-tools and legacy adapters
- `grep`: candidates are built from the lexical requested path, so a relative deny on a symlink alias applies while absolute rules still apply to the canonical target
- tests: shadowed DELETE / Mongo write pipeline rejected before any ask or redirect; four Binary normalisation cases with real BSON objects; six absolute-ask cases across grep/glob/ls through the real permission service (reject and once); symlink alias deny vs real path
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* fix(analyst): report a refused `sql_materialize` statement as an error result before routing
Moving the read-only validation ahead of the permission ask and workspace redirect made a refused statement throw out of `execute()`; the tool contract (and the existing `sql-materialize.test.ts` case) is an error result. The early refusal now returns the same `Materialize: ERROR` result shape, with no redirect annotation, whether or not the warehouse is shadowed. The routing tests assert the result instead of a throw.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* fix(analyst): approve search scope before reading, deny on alias and target, raw BSON bytes, read-only PostgreSQL checks, DuckDB state-loss disclosure, hard-deadline finalization
- `grep`: candidates are listed first (`--files`, bounded), read approval is requested for ask-scoped files before any content is read, and the content search then runs only over approved candidates; a candidate is denied when either the requested alias path or the resolved target is denied. When nothing is restricted the search is a single ripgrep run as before; when some candidates are denied or unapproved, ripgrep is restricted to the approved files in bounded argument batches so a denied file with many matches is neither read nor allowed to consume the scan ceiling
- Mongo driver: BSON `Binary` is encoded from raw bytes (`value(true)`), so bytes at or above `0x80` are no longer re-encoded as UTF-8 before base64
- `sql_check`: PostgreSQL checks execute with `readOnly: true`, as materialization does, so a `SELECT` over a DML user-defined function cannot mutate data
- DuckDB driver: the shared engine tracks a generation; after one connector's timeout restarts the helper, every other connector sharing it is rejected once with `SESSION_STATE_LOST` before executing more SQL, including a peer already queued, then continues on the new generation
- `run`: a text part forcibly closed by the soft-budget interruption no longer counts as an answer, and hard-budget exhaustion always enters the bounded finalization once
- tests: target-deny behind an allowed alias; approval precedes any read, rejection reads nothing, newly denied candidates excluded; high-bit Binary bytes round-trip (`00 80 ff` → `base64:AID/`) on both BSON value APIs; PostgreSQL `readOnly` passed and a write-performing SELECT rejected; peer state-loss disclosed once (next call and already queued); partial-text stall gets one finalization while a complete answer skips it
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* fix(analyst): a grep listing cut at the scan cap never takes the unrestricted search path
When `--files` listing stops at the 10,000-path cap, every listed candidate can still be approved while unlisted files remain, so the single unrestricted search could read a denied unlisted file before `filterPath` rejected it. The fast path now also requires the listing to be complete; a capped listing always searches only the approved files.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* fix(analyst): disable the shell for the read-only analyst and keep the caller's Azure CLI login for DuckDB helpers
- The analyst and its data-explorer subagent no longer get `bash`: the tool evaluates individual parsed command nodes, so pipes, chaining and redirects (`cat /dev/null > model.sql`, `find . -delete`, `(cat x)>y`) were invisible to the read-only allow-list and ran without approval. Bash is now denied for both agents, also after global or per-agent configuration grants; structured read, grep, glob and SQL tools remain. Simple shell reads (`ls`, `cat`, `find`, `dbt list`) are no longer available to these agents.
- Ordinary DuckDB helpers now receive `AZURE_CONFIG_DIR` (the caller's explicit value or `$HOME/.azure`) so the Azure `credential_chain` `cli` provider finds the user's login although `HOME` points at the private helper directory; scratch helpers stay isolated. The scratch/ordinary distinction is threaded into initial spawn and respawn.
- tests: shell writes, chaining and substitutions denied for both agents and under config grants; Azure config directory present for ordinary helpers (default and explicit), absent for scratch, across startup and respawn
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* fix(analyst): identical glob semantics for the grep preflight and content pass; reject every statement a peer queued before a DuckDB session restart
- `grep`: an explicit `include: "*"` is normalised away for both passes, so the `--files` listing and the content search see exactly the same files and no unlisted file is read on the unrestricted path
- DuckDB driver: the generation seen when a statement is queued is compared with the engine generation at dispatch, so every statement a peer queued before the restart is rejected with `SESSION_STATE_LOST` (a queued `BEGIN`/`INSERT` pair can no longer half-execute in autocommit); an unacknowledged peer is still rejected once on its next call, and the connector whose own timeout caused the restart keeps its queued statements, as before
- test: `rejects every statement a peer queued before the shared-session restart`
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* fix(drivers): keep the caller's home and cloud credential caches visible to ordinary DuckDB helpers
Ordinary helpers had HOME, USERPROFILE and every XDG directory redirected to the private helper directory, which hid file-backed credential caches that worked in-process: the AWS IAM Identity Center token cache under ~/.aws/sso/cache (resolved from HOME) and gcloud Application Default Credentials under $CLOUDSDK_CONFIG or ~/.config/gcloud. Bun script execution does not load the global bunfig (verified on the installed Bun: temp HOME/XDG configs with observable preload/define settings were ignored under the explicit --config and --no-env-file the helper already uses), so the redirect protected nothing for ordinary helpers. They now inherit the parent's HOME, USERPROFILE and XDG_* when defined, plus CLOUDSDK_CONFIG (explicit or <home>/.config/gcloud) and AZURE_CONFIG_DIR; scratch helpers keep the private directory for all of these. TMP variables stay private.
tests: provider filtering with preserved home; default and explicit Azure and gcloud directories for ordinary helpers; scratch helpers private; four startup/respawn combinations.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* test(drivers): align the helper-environment review tests with the ordinary/scratch home contract
Ordinary helpers now keep the caller's HOME (scratch helpers keep the private directory); four review-suite assertions still expected the private home for ordinary helpers and failed the Driver E2E job. They now assert the caller's home for ordinary helpers and the private home for scratch helpers.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* fix(analyst): read-only transactions for every PostgreSQL read path, cancel queued pool acquisitions at the deadline, Windows gcloud default
- Every read path that executes against a PostgreSQL connector now passes `readOnly: true`, as `sql_check` and materialization already did: value profiling in `schema_inspect`, parsed `sql_execute` reads, `sql.execute`/`sql.explain` dispatch (incl. EXPLAIN ANALYZE), the connection-test probe, data-diff discovery and diff queries, and FinOps query history. A view whose SELECT calls a data-modifying function can no longer mutate data under read permission.
- The outer `query_timeout_ms` deadline aborts a PostgreSQL acquisition still waiting on the pool; a client admitted after the deadline is released without issuing SQL, and cancellation is checked again before user SQL, so a statement can no longer execute after the caller received a timeout.
- DuckDB helpers default `CLOUDSDK_CONFIG` to `%APPDATA%\gcloud` on Windows (home-based fallback when APPDATA is unset); Unix default unchanged.
- tests: readOnly asserted per call site for postgres/postgresql with non-PostgreSQL controls; queued acquisition abandoned and late client released without SQL, admitted client executes; Windows and Unix gcloud defaults
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* fix(drivers): one restart-generation guard in LocalEngine for SQLite and DuckDB; read-only requests bypass dbt; SAP Claude reasoning cleanup
- `LocalEngine` now owns the FIFO queue and the helper generation: every statement enqueued before a helper restart (timeout or crash) is rejected with `SESSION_STATE_LOST` / `stateLost: true`, including the statements of the connector whose own timeout caused the restart (its timeout error already disclosed the loss); each other connector sharing the helper is disclosed to once on its next call; fresh statements then run. SQLite gets the guard it lacked; DuckDB's separate implementation is removed. Structured error codes and flags survive dispatch.
- `read_only` requests bypass the dbt execution path and use the native PostgreSQL connector with `BEGIN READ ONLY` … `ROLLBACK`; a request with only a dbt profile fails before any SQL runs rather than executing unprotected.
- `budgetMessages` strips reasoning parts for Claude served through `@jerome-benoit/sap-ai-provider-v2` (the SAP converter cannot round-trip Anthropic signatures), dropping an assistant left empty; non-Anthropic SAP history is unchanged.
- tests: SQLite and DuckDB stale-queue rejection with and without an open transaction, peer disclosure (next call, already queued), crash disclosure, byte-cap and structured error preservation, spill cleanup after one disclosure; read-only PostgreSQL requests skip dbt and are protected; SAP reasoning removal
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* fix(analyst): reject every pre-acknowledgement call after a helper restart, validate MongoDB bounds, honor replace mode for empty results
- `LocalEngine` captures the session's acknowledged generation at enqueue as well as the engine generation; every call a session enqueued before it acknowledged the current helper generation is rejected with `SESSION_STATE_LOST`, not only the first. Calls enqueued after the acknowledgement run.
- `sql.execute` with `read_only: true` is refused for any non-PostgreSQL warehouse, named or default, before any SQL executes.
- MongoDB `find`/`aggregate`/`countDocuments` bounds are validated before a cursor is opened: `limit` must be a non-negative safe integer (zero keeps the default cap), `skip` non-negative, pipeline `$limit` positive and `$skip` non-negative, for the string, object and explicit command forms. A negative `limit` can no longer turn into an unbounded cursor.
- `sql_materialize` in `replace` mode with an empty, column-less source now drops the existing scratch table and reports it instead of leaving stale rows behind; `append` with an empty source preserves the rows and reports their count.
- tests: three back-to-back peer writes after a restart all rejected (SQLite and DuckDB); named Snowflake-type warehouse refused for read-only; MongoDB bound rejection per entry form and before cursor open; replace/append with empty sources; the deadline test restores the real `pg` module after its suite so the connection-probe test in another file no longer sees the queued fake pool (CI `TypeScript` failure on the previous two heads).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* fix(analyst): dispatch notification for helper calls, distinguish a `"NULL"` string from SQL NULL, check create conflicts before empty results
- `LocalEngine.call` accepts an `onDispatch` observer that fires once after a call's message is flushed to the helper process (never for calls rejected while queued); DuckDB exposes it as `ExecuteOptions.onDispatch`. The scratch shutdown test waits on it with a bounded 5 s timeout instead of a fixed 50 ms sleep, so it exercises the active-query close path deterministically.
- `sql_execute` previews render a string cell whose text is exactly `NULL` as `"NULL"`; a database null stays bare `NULL`. All other cells are unchanged; the JSONL spill already distinguished the two.
- `sql_materialize` checks the `create`-mode conflict with an existing scratch table before the empty-result early return, so an empty source can no longer leave a stale table in place under the default mode.
- tests: dispatch observer fires once per dispatched call and never for rejected queued calls; scratch close under a dispatched query (5 consecutive runs, 0 flakes); `NULL | "NULL" | null | x` and array-cell rendering; default create with an empty source refuses an existing table and preserves its rows.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* fix(analyst): keep SQL preview cells unambiguous — quote delimiter-bearing strings, render structured values as JSON
- A string cell that contains a newline, a carriage return or the column separator ` | ` (or equals `NULL`) is rendered through `JSON.stringify`, so one database row can no longer appear as two rows or an extra column in a preview that fits without a spill; inside comma-joined array cells a string containing a comma is quoted too. Every other string is unchanged.
- Plain objects (DuckDB STRUCT/MAP, PostgreSQL JSON/JSONB, objects nested in arrays) render as compact JSON instead of `[object Object]`, bounded by the existing cell cap and clip marker, with the array path's cycle and depth guards; `bigint` inside an object renders as its decimal string. Date, top-level bigint, bytes, numbers and booleans keep their paths. The JSONL spill already preserved both cases.
- tests: delimiter and carriage-return quoting, comma quoting in arrays, distinct object cells, nested objects, self-reference and deep nesting clip without hanging, oversized object clipped and spilled, bigint in objects, preserved special-value paths, JSONL spill preservation.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* fix(analyst): object preview cells serialize in one guarded pass
- `displayObject` converts `bigint` in a single `JSON.stringify` pass and catches any serializer failure (a throwing `toJSON` or getter), rendering `…[cell not serializable]` and marking the preview clipped instead of aborting the whole result; the previous second pass was unguarded.
- test: a throwing getter and a throwing `toJSON` degrade to the placeholder while sibling cells render.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* fix(analyst): binary preview cells as base64, delimiter-bearing aliases quoted, private-trace test waits for delivery
- Every `Uint8Array`/`Buffer` cell renders as `base64:<data>` (matching the JSONL spill's `$bytes` tag), encoding only a bounded prefix, so distinct byte payloads no longer collapse to the same decoded text; the oversized path keeps its clip marker.
- A column alias containing a newline, carriage return or ` | ` is JSON-quoted before width accounting, so one real column cannot appear as several headings.
- The private-trace bus test awaits the subscription's own delivery signal (bounded at 5 s) instead of a 10 ms sleep.
- tests: distinct small buffers, Uint8Array views and nested binary cells, base64 expansion under the threshold, oversized prefix; separator and newline aliases, alias clipping after escaping.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* fix(provider): the stream idle watchdog measures idleness at the model-output boundary
- The watchdog kept a per-read timer that every successful read reset, so a provider emitting only protocol keepalives (SSE comment or blank lines, Amazon event-stream heartbeat/ping/keepalive/metadata frames) never timed out. It now keeps one monotonic deadline that only model output extends: a completed SSE `data:` line (field prefixes buffered across partial chunks) or an event-stream frame whose `:event-type` is not a heartbeat type; unparseable framing or headers count as output so nothing can hang on a parser gap. Protocol-only bytes are still forwarded and still mark the stream as emitted; an already-expired deadline fails immediately.
- tests: SSE keepalives time out while data at the same cadence does not; split and incomplete data lines; event-stream heartbeat/ping/keepalive/metadata vs chunk frames, typed headers, split and coalesced frames, malformed prelude and header.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* fix(analyst): SSE ping payloads do not extend the idle deadline; bounded Mongo distinct and aggregates; PostgreSQL timeout cache invalidation
- The idle watchdog now tracks whole SSE event blocks: a block counts as model output only when it has a `data:` line and its `event:` is not `ping`/`keepalive`/`heartbeat`, and an event-less `{"type":"ping"}` payload is excluded, so a stalled Anthropic stream that keeps pinging still times out. Only a 64-byte line prefix is ever retained. Amazon event-stream `chunk` frames extend the deadline as soon as their headers are parsed and on each payload piece.
- MongoDB `distinct` runs as `$match`/`$group`/`$limit cap+1` instead of materialising the full value array; `aggregate` appends a terminal `$limit cap+1` unless the last stage already is a `$limit` within the cap, so an earlier `$limit` followed by an expanding stage can no longer materialise unbounded rows.
- The PostgreSQL per-client timeout cache is invalidated whenever user SQL mentions `statement_timeout`, so a `SET`/`RESET` by a permitted statement makes the next call re-read the session value and apply our bound.
- tests: ping/keepalive/heartbeat events with data and event-less ping payloads time out while content events and long real data reset; CRLF and BOM splits; slow event-stream chunk pieces; bounded distinct with truncation; terminal cap after `$limit` + `$unwind`; preserved small terminal `$limit`; stricter cached timeout kept for ordinary queries and rechecked after SET/RESET.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* fix(drivers): `RESET ALL` and `DISCARD ALL` also invalidate the cached PostgreSQL statement timeout
- Both statements clear every session setting including `statement_timeout`, so the per-client cache is dropped for them too; the next call re-reads the session value and applies our bound.
- tests: the recheck regression now also covers `RESET ALL` and `discard all`.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* fix(drivers): bounded MongoDB distinct keeps per-element array semantics
- The bounded `distinct` grouped array-valued fields by the whole array and collapsed missing fields with explicit null; an `$unwind` ahead of the `$group` restores the native `distinct` semantics (array elements reported individually, missing fields skipped) while keeping the `cap + 1` bound.
- tests: both distinct pipeline expectations include the unwind stage.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* fix(analyst): lossless timestamp previews, null array items marked, explicit nulls kept in bounded Mongo distinct
- `Date` cells render as ISO 8601 (`toISOString`) instead of the default string form, which dropped milliseconds and the zone; matches the JSONL spill's `$date`.
- A null array element renders as `NULL` instead of an empty string, so `[NULL]`, `['']` and `[[]]` stay distinguishable in a preview that does not spill.
- The bounded Mongo `distinct` excludes missing fields and empty arrays with a `$match`, then unwinds with `preserveNullAndEmptyArrays`, so explicit `null` values are reported as the native command reports them.
- tests: ISO timestamp with milliseconds; `[null, "NULL"]`, `[""]` and `[[]]` rendering; both distinct pipeline expectations.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* fix(analyst): bound the initial PostgreSQL timeout probe; a late soft reminder interrupts only its own generation
- The first `SHOW statement_timeout` on a freshly acquired pooled client ran without any deadline, so a server that accepts the connection but stalls on its first statement hung the call past the caller's deadline. The probe now races the caller's abort signal (or the resolved query deadline when no signal is given); on abandonment the exact abort reason propagates, no user SQL runs, and the client is discarded instead of being returned to the pool (pg's promise API has no cancellation handle).
- A soft reminder captured whichever generation was current when its prompt resolved, so a reminder that outlived the CLI's client timeout could interrupt the hard-deadline finalization that had since started. The interruption is captured when the reminder is issued and fires only if that generation is still current.
- tests: stalled probe abandoned with the exact caller and PostgreSQL-shaped abort reasons and at the resolved deadline without a signal; fast probe unchanged; late soft and soft-no-downgrade reminders leave a newer generation alone while a still-current one is interrupted.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* fix(analyst): bound the PostgreSQL timeout probe by the effective deadline; order the queued-call test on dispatch
- The initial `SHOW statement_timeout` probe now uses the effective deadline
(tighter of `statement_timeout` and the query timeout) for its timer and its
`QueryTimeoutError`. A configured `statement_timeout` with the query timeout
disabled no longer leaves the probe unbounded.
- The SQLite connector forwards `onDispatch` to the local engine, as DuckDB does.
- The queued-call deadline test waits for the first query's dispatch instead of
a fixed sleep.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(analyst): keep the probe deadline beside a caller signal; distinct empty arrays; read-only `EXPLAIN ANALYZE`
- The initial `SHOW statement_timeout` probe always arms the effective deadline,
capped at the timer limit, alongside any caller signal. A signal that fires
late or never no longer extends the probe, and a `statement_timeout` above
2^31 ms no longer overflows into an immediate timeout.
- Result previews render an empty array as `[]`, quote empty strings inside
arrays and quote a literal `[]` string, so `['']`, `[[]]` and `""` differ.
- `sql_explain` with `analyze: true` refuses anything but a single read-only
query before dispatch. EXPLAIN ANALYZE executes the statement, and engines
without a read-only transaction would otherwise apply a write. Plans without
`analyze` are unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(analyst): bound the `SET statement_timeout` setup query like the probe
Both session setup queries now go through one helper that races the query
against the effective deadline (capped at the timer limit) and the caller signal,
and discards the client when it abandons one. A stalled `SET` can no longer pin
a pooled client.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(sql-materialize): read each source once outside the scratch lock (#1515)
* fix(sql-materialize): read each source once outside the scratch lock
- Read and buffer the source before entering the scratch connection lock, so a slow remote read no longer blocks unrelated scratch operations in the same root session.
- Remove the preflight count and sample queries; execute exactly one bounded source statement (`max_rows + 1`, driver byte cap, defensive row-byte check) and reject truncated results before touching the destination.
- Enforce a single remote source deadline without the extra grace period, aborting `PostgreSQL` pool admission on expiry; late results never reach the write phase.
- Keep `MongoDB` aggregate terminal limits and read-only protections.
- Add concurrency regression tests; update materialize, routing and `MongoDB` tests for the single-read behavior.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(sql-materialize): keep scratch-to-scratch derivations atomic
- When the source is the scratch connection itself, read and write under one scratch-lock acquisition so concurrent derivations from the same table cannot lose updates.
- Remote sources still read outside the lock.
- Add a regression test for concurrent self-derived replaces.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(sql-materialize): bound aggregate buffering across overlapping remote reads
- Reserve each remote read's byte cap from a process-wide buffer budget before reading outside the scratch lock; a read larger than the budget runs alone.
- Add a regression test that two large overlapping reads do not run together.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* style: format `sql-materialize.ts`
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(sql-materialize): hold the buffer reservation until the scratch write finishes
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* fix(session): normalize BigInt tool results before persistence (#1518)
- Add opt-in `safe-number` mode to `normalizeBigInts`: safe-range BigInts become numbers, larger values exact decimal strings; defaults unchanged
- Normalize parts once in `Session.updatePart` (legacy) and the Effect-service `updatePart` before DB write, event publish and return
- Add regressions: real prompt loop with concurrent `sql_execute` calls carrying nested BigInt metadata, migrated service/projector test, helper boundary tests
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* fix(sql): bound query deadlines and propagate session cancellation (#1519)
* fix(sql): bound query deadlines and propagate session cancellation
- per-call and env zero timeouts fall through to the configured/default policy; only connection config can disable the deadline
- add a shared `executeQuery` boundary combining session cancellation, the configured deadline and a run-level query cutoff
- send one absolute query cutoff with work requests (reserving answer time before the hard work deadline); child tasks inherit it
- forward session abort through `sql_execute`, `sql_check`, `sql_materialize`, `sql_explain` and value profiling
- close MongoDB cursors on abort/deadline and stop fetching; discard active PostgreSQL clients on abort
- cancel queued and active SQLite/DuckDB work; an aborted active call reports `SESSION_STATE_LOST` on the next statement
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(sql): …
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Issue for this PR
Closes #1424
Type of change
What does this PR do?
Why. A session never recovers when a model request goes quiet. In a recorded benchmark, 10 of 135 sessions on one model (GPT-6.1 Sol via Bedrock Mantle) logged a request and then nothing until the harness killed them at its 30 minute limit: no error, no retry, and in 7 of the 10 the finish-time validators never ran. (10 of 135: estimate from recorded data.)
Root cause (verified in code).
What changed (before, after).
Spec. A stalled request is aborted and retried with the existing backoff at most 5 times; partial output is never duplicated; a dispatched tool is never re-run; exhaustion ends with a clear error; cancel is immediate.
Tenant/user impact. A request silent for 300s is now retried instead of hanging. Users could also notice: (1) a persistent stall takes about 31 minutes to surface at the default (6 attempts of 300s plus backoff), so lower headerTimeout where that matters; (2) the 300s first-byte deadline now also bounds custom fetches such as Vertex credential acquisition; a healthy request silent for over 300s before headers (slow local prompt evaluation, non-streaming calls) would be cut off, so raise headerTimeout there; (3) the no-retry-after-started-tool rule covers every retryable error: a mid-stream 5xx after a tool started used to be retried (re-running the tool) and now ends with an error, and, as for any errored step, its tool history is not replayed next turn; (4) the TUI shows "Model stopped responding" for the exhausted error; (5) a Stop just before a backoff ends the wait at once. Validators are unchanged: they run after a clean stop, so a recovered stall reaches them; an exhausted one skips them.
Deployment readiness. No migration, dependency or config; revert restores old behaviour.
How did you verify your code works?
Screenshots / recordings
Not a UI change.
Checklist
🤖 Generated with Claude Code
Note
Medium Risk
Changes core provider fetch timeouts, session retry semantics, and tool-call safety rules; mis-tuned
headerTimeoutcould cut off legitimately slow streams, and the no-retry-after-tool-start rule alters behavior for any retryable error mid-step.Overview
Sessions that hung or died on silent providers now abort stalled requests, retry with backoff (up to 5 times), and clean up partial output so retries do not duplicate streamed text or re-run tools.
Provider layer: Most providers get a 300s default
headerTimeout(configurable viaoptions.headerTimeout, orfalseto disable). The stream idle watchdog now covers Amazon Bedrock event streams as well as SSE. Custom fetches that ignore abort signals (e.g. Vertex credential work) are raced against the first-byte deadline.Session layer: Header and idle-stream timeouts map to retryable "The model stopped responding" errors. Before each retry,
StallRecoveryremoves text, reasoning, step-start, and pending tool parts from the failed attempt; retries are blocked once a tool has started executing or a step has finished, with a clear error message. User cancel skips retry and fails fast during retry backoff if the signal is already aborted.Docs / TUI:
providers.mddocumentsheaderTimeoutandchunkTimeout; notifications treat the new exhausted-stall message as "Model stopped responding".Reviewed by Cursor Bugbot for commit a689e5a. Bugbot is set up for automated code reviews on this repo. Configure here.