Context limits are treated as a hard overflow boundary, not a quality gradient; and provider prompt caching is never used #2341

Summary

Two related findings about how PDD handles large assembled contexts.

1. Context is modelled as a cliff, not a slope. Every mention of context limits in the docs and code treats overflow as the failure mode. Published long-context research shows quality degrades substantially well before the limit, so a prompt PDD reports as healthy at 60% of the window may already be generating materially worse code.
2. PDD never uses provider prompt caching. cache_control appears zero times in the codebase. Response-level memoization is used instead, which is a different mechanism and misses a large cost and latency saving on exactly the workload PDD generates.

Inspected at commit 817abe2. Static inspection only; no PDD commands were run and no LLM calls were made.

Evidence — context treated as overflow-only

Overflow framing (README.md:3794):

Prompt exceeding model context window — PDD validates prompt token count before sending to the LLM. If the prompt exceeds the model's context limit, PDD reports the token count, the model's limit, usage percentage, and which prompt caused the overflow. When multiple candidate models are configured, PDD automatically falls back to the next model.

The remedy is a bigger window (pdd/llm_invoke.py:5420):

                litellm_kwargs["extra_headers"] = {"anthropic-beta": "context-1m-2025-08-07"}

The README likewise advises using "a model with a larger context window (e.g. Gemini with 1M tokens, or Claude which automatically uses the 1M beta header)".

Compression is framed as cost, not quality (docs/prompting_guide.md:1745):

To manage large context windows and reduce costs, PDD supports automated context compression.

Nothing in the repo discusses degradation. Searching all *.md and *.py for context rot, lost in the middle, or degrad.*context returns zero matches.

The measurement tooling already exists. pdd context reports per-source token attribution with a default 80% threshold and exit code 2. pdd/fix_focus.py:33-34:

LARGE_CODE_THRESHOLD = 500    # lines in code file
LARGE_TEST_THRESHOLD = 1000   # lines in test file

and pdd/compressed_sync_context.py:14 sets DEFAULT_BUDGET = 12000. So thresholds, budgets, and per-source attribution are all present — the framing just attributes them to cost and overflow rather than to output quality.

Why this matters here specifically. A PDD prompt is assembled from the prompt body plus auto-deps includes plus an example plus accumulated tests — several artifacts describing the same behaviour in overlapping terms. Published results indicate semantically similar distractors are the strongest driver of degradation, and that degradation is continuous rather than a cliff:

• NoLiMa: 11 models fall below 50% of their short-context baseline at 32K; GPT-4o drops from 99.3% to 69.7%.
• Chroma, Context Rot: all 18 frontier models tested degrade continuously as input grows, with distractor similarity dominant; coding agents named as a primary affected case.
• RULER: most models score best at the shortest setting tested.

Under those findings, "fall back to a 1M-token model" resolves the overflow error while moving the prompt further into the degraded regime.

Evidence — no provider prompt caching

$ grep -rn "cache_control" pdd/ --include='*.py' | wc -l
0

The only anthropic-beta header used is context-1m-2025-08-07 (llm_invoke.py:5420); no cache_control breakpoints are ever set. What PDD uses is LiteLLM response caching — whole-response memoization keyed on the request (llm_invoke.py:1343-1410 configures S3/GCS with SQLite fallback; :5770 sets litellm_kwargs["caching"] = True).

These are different mechanisms with different economics. Response caching only helps on an exact repeat of the same call. Provider prompt caching discounts the shared prefix across calls that differ in their tails — precisely the shape of PDD's workload: sync runs auto-deps → generate → example → crash → verify → test → fix → update, and agentic workflows run 11-18 steps over largely overlapping context. Every step currently pays full prefill price.

--one-session exists because multi-session runs are expensive (the README describes it as producing "faster and cheaper sync runs"). Prompt caching attacks the same cost from a complementary direction and would compose with it.

Proposed changes

1. Report context usage as a quality signal, not just an overflow guard. pdd context already computes what is needed; add a warning band (e.g. above ~40-50% of window) noting generation fidelity may be degraded, and surface it during sync.
2. Prefer splitting over a bigger window. When a prompt approaches the degraded regime, recommend pdd split rather than silently escalating to a 1M-context model.
3. De-duplicate overlapping context. auto-deps already dedups inline content that duplicates included documents; consider extending that to the prompt/example/test overlap, which is the classic distractor pattern.
4. Adopt provider prompt caching. Add cache_control breakpoints for the stable prefix (system/instruction blocks, then long includes) on providers that support it, and order assembled prompts stable-prefix-first to maximise the cacheable region. Note this interacts with <shell>/<web> tags, which mutate prompt bytes — another argument for placing dynamic content last.
5. Publish measured fidelity-vs-context data. PDD is unusually well positioned: 181 prompt/code pairs, an accumulated test suite, and cost instrumentation already in place. Regeneration pass-rate as a function of assembled context size would let adopters size dev units on evidence.

Item 4 is likely the largest immediate cost win available in the codebase; item 1 is the largest quality win for the least code.

Happy to open a PR for item 4 if the approach sounds right.