Skip to content

Compact the conversation when it nears the model's context window - #28

Merged
vicgupta merged 1 commit into
mainfrom
issue-24-compact-context
Oct 24, 2026
Merged

vicgupta merged 1 commit into
mainfrom
issue-24-compact-context

Conversation

@vicgupta

@vicgupta vicgupta commented Oct 24, 2026 •

Copy link
Copy Markdown
Owner

Closes #24

What changed

  • When: before each request, tinyagent estimates its size (the last request's reported input + output tokens, plus ~4 chars/token for messages added since). Above 80% of the context window it compacts proactively. If a provider rejects a request with prompt is too long, context_length_exceeded or maximum context length, it compacts and retries that step once; a second overflow ends the run as Error (context still exceeds the window after compaction: ...).
  • Window size: new model.ContextWindows table, longest prefix wins (gpt-4o 128K, gpt-4.1* 1,047,576, claude-* 200K), also matching nested names like openrouter/anthropic/claude-*. New config key context_window overrides it. With neither, only the reactive path runs.
  • How (internal/agent/compact.go):
    1. Elide old tool outputs to [output elided: N chars], outside a protected zone: system prompt, task, and the recent turns (~25% of the window, never fewer than the last 2 assistant turns with their results). Stops if the estimate drops under 50% of the window (under half the previous estimate when the window is unknown).
    2. Otherwise one summarization request (no tools) gets a text transcript of the older turns and asks for Task / Constraints / Progress so far / Files changed / Key decisions / Open problems / Next steps. The model then sees system + task + a clearly marked summary message + the protected turns.
    3. Still over 80% of the window, or nothing could be compacted: the run ends as Error. No loops.
  • Invariants: the protected zone always starts at an assistant message, so tool calls and results never split. The agent keeps a separate view of the history for the model; Trajectory.Messages stays the full original history.
  • Trajectory: new compactions (omitted when empty), one record per compaction: step, reason, stage, tokens before/after, elided and summarized message indexes, summary text, summary cost, and view (the message indexes the model is sent next, with -1 for the summary).
  • Accounting: the summary call's tokens and cost go into the run totals and --cost-limit is checked right after compaction; it doesn't count as a step.
  • Output: one stderr line per compaction, e.g. compacted context: ~17,020 → ~9,048 tokens (elided 4 tool outputs).
  • --no-compact (in the Limits help group) turns it all off; requests are then exactly the full history, as before.
  • README: flag table, context_window in the config example, and a new "Long runs" section (80% threshold, stages, lossy summaries, put must-hold rules in AGENTS.md).

Testing

  • gofmt -l clean, go build ./..., go vet ./... clean, go test -race -count=1 ./... passes in every package (Go 1.27.3).
  • New tests in internal/agent/compact_test.go, using a fake model and real read_file calls:
    • Elide-only: over 80% of a configured context_window (overriding the 128K table value for gpt-4o), the next request has steps 1–4 elided, steps 5–6 verbatim, system and task unchanged, no summary call; trajectory keeps original outputs and records the compaction; stderr line checked.
    • Summarize: exactly one tool-less summary request; next request is system + task + summary + recent turns (including an assistant turn with 3 tool calls); totals include the summary tokens/cost but not a step; the saved -o JSON has the compaction record.
    • Summary cost trips --cost-limit.
    • Reactive: Anthropic- and OpenAI-style overflow errors with no known window trigger one compaction and one retry, and the run submits. A second overflow ends the run as Error after exactly one retry.
    • --no-compact: every request equals the full history, no stderr output, overflow ends the run as Error.
    • No window: no proactive compaction.
    • Tool call / result pairing is checked on every request, and the protected-zone boundary is tested directly across several sizes.
  • internal/model/context_test.go covers the window table and overflow detection.
  • Not tested against live provider APIs (no keys in the sandbox).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Compact the conversation when it nears the model's context window

1 participant