A local-first AI coding agent for VS Code — where your compiler, not the model, decides what ships.
Caveman Code runs entirely against Ollama on your own machine — no cloud, no API keys, no telemetry.
Ask it questions about your codebase, or describe a change and review the diff before anything touches disk.
- Chat side bar with streaming answers, rendered markdown, and collapsible model reasoning.
- Agentic loop — the model can read files, search the workspace, list directories, and run shell commands (with your approval), then keep working from the results.
- Reviewable edits — every change is staged as a diff card with Apply / Reject / Open diff.
Nothing is written until you say so (unless you turn on
caveman.autoApplyEdits). - Workspace index — file paths, languages and symbol names are indexed so the agent starts with relevant context instead of guessing.
- Project rules — a
.cavemanrulesfile in your workspace root is injected into every system prompt, and compiled into constraints the Forge scorer enforces. - Caveman Forge — several local models compete on the same task, each candidate is compiled and linted in a shadow copy of your project, and your toolchain picks the winner. The leader then repairs itself from its own compiler errors.
- Time travel — every write is checkpointed, so any change is rewindable later.
-
VS Code 1.85 or newer
-
Ollama running locally (
ollama serve) -
At least one chat model:
ollama pull qwen3.5:9b
Any instruction-following model works. Larger models follow the edit protocol more reliably;
qwen3.5:9bis a good default on 8 GB+ of VRAM.
npm install
npm run compileThen press F5 in VS Code to launch the Extension Development Host, or package it:
npx vsce package
code --install-extension caveman-code-0.3.0.vsixOpen the Caveman icon in the activity bar, or press Ctrl+Shift+Alt+C.
Small local models are bad at producing unified diffs — they get line numbers wrong almost every time. Caveman Code does not ask for one. Instead the model emits SEARCH/REPLACE blocks:
```caveman-edit src/server.ts
<<<<<<< SEARCH
const port = 3000;
=======
const port = Number(process.env.PORT ?? 3000);
>>>>>>> REPLACE
```
The SEARCH text is matched against the file by content, through a ladder of strategies:
- exact substring match
- line match ignoring trailing whitespace
- line match ignoring indentation, re-indenting the replacement to fit
- fuzzy window match, only above an 82% similarity threshold
A fuzzy match must also keep the block's identifiers: export function subtract(a)
is over 90% character-similar to export function add(a), so similarity alone would
happily rewrite the wrong function. At least 80% of the identifiers in the SEARCH text
have to be present in the matched window, which is what stops that.
The resulting file content is computed locally and applied as a single whole-document write, so there is no line arithmetic to get wrong.
Small models reliably make one specific mistake: they put SEARCH/REPLACE pairs for several files into a single block addressed to the first one. Rather than rejecting the whole block:
- the hunks that did match are still applied, so the model does not have to redo them;
- each hunk that failed is looked up across the workspace, and if its text lives in
another file the model is told exactly which one — "Change 2 is not in
src/math.ts, that text is insrc/totals.ts. Put it in its own block." - only genuinely unlocatable hunks fall back to "here is the closest text in the file".
Before this, a model that made that mistake would loop on it until it ran out of steps.
After it, the same model finishes the same task in three turns. scripts/forge-trace.mjs
prints every turn a model takes and is how that gets diagnosed:
$ node scripts/forge-trace.mjs qwen3.5:9b
── turn 1 ── actions: Search "add"
── turn 2 ── actions: Edit src/math.ts | Edit src/totals.ts | Edit src/report.ts ...
── turn 3 ── actions: (none) >>> no actions: the loop ends here
turns: 3 edits: 3 file(s) elapsed: 23.6s
The same fenced-block protocol covers the other actions:
| Block | Purpose |
|---|---|
caveman-read <path> |
Read a file into the conversation |
caveman-search |
Search the workspace (body is the query; /regex/ supported) |
caveman-list <dir> |
List a directory |
caveman-edit <path> |
SEARCH/REPLACE edit |
caveman-create <path> |
New file, body is the full contents |
caveman-delete <path> |
Delete a file (moved to trash) |
caveman-run |
Shell command, subject to approval |
A 9B model cannot reliably one-shot a correct edit. Forge stops treating the model as an oracle and starts treating it as a search process, with your own toolchain as the referee.
Press the Forge button in the composer (or Ctrl+Shift+Alt+F) and one request becomes:
-
An ensemble proposes. Several local models answer the same question independently. Different architectures make different mistakes.
-
Each candidate lands in a shadow workspace — a real mirror of your project in the temp directory. Your files are never touched.
-
Your real toolchain judges it. Whatever the project already uses:
tsc, ESLint,cargo check,go build, Python syntax, and optionally your test suite. Nothing is installed or downloaded; only tools already present are run. -
Candidates are scored on what they broke, what they fixed, whether they obey
.cavemanrules, and how small the change is. -
The leader repairs itself. The winning candidate is handed its own compiler errors and tries again — and files named in those errors are pulled into its context automatically. The compiler turns out to be a far better retrieval signal than keyword search.
Three guarantees hold here: a repair that scores worse than the state it started from is rolled back (candidate, overlay and conversation together); a round that answers without editing anything is retried once, more forcefully, rather than burning the budget; and a candidate is never left mid-phase, so nothing spins forever in the race view.
-
Only the winner is shown to you, as a normal reviewable diff — and if even the best candidate still does not compile, the UI says so above the diff in red, listing the remaining errors. Forge offers the best candidate it found, which is not always a working one, and a diff that looks reviewed must never imply it was verified.
Here is a real run against a local ensemble, on the classic failure mode — a rename whose call sites live in files the model was never shown:
$ node scripts/forge-demo.mjs qwen3.5:9b
· referees: TypeScript
· baseline: 0 error(s), 0 warning(s) in 1566ms
· qwen3.5:9b: score -104 (1 new problems, 0 fixed, +4/-4)
· qwen3.5:9b: after repair 1, score 495 (0 new problems)
CANDIDATE VERDICT CHURN SCORE TIME
------------------------------------------------------------------------------
qwen3.5:9b compiles clean [repaired x1] +5/-5 495 29.1s
WINNER: qwen3.5:9b — toolchain clean
files: src/math.ts, src/totals.ts, src/report.ts
source: UNCHANGED on disk (nothing applied without review)
The model made exactly the mistake a 9B model makes — it renamed the definition and
missed a caller in a file it had never been shown. tsc caught it, the repair round
fixed it, and what reached the review UI compiled. One 9B model, 29 seconds, verified.
That single-model run is the sharper demonstration: the ensemble helps, but the loop is what turns an unreliable model into a reliable one.
Candidates are compiled somewhere else entirely, because verification means running real tools that want a real directory tree. Editing your files and undoing them would fight your editor, your file watchers and your git status — and leave you broken if anything crashed mid-verify.
The mirror is persistent and synced incrementally: the first sync copies your source
files, every later one is a stat walk. Heavy directories (node_modules, target,
.venv) are linked, not copied — a directory junction on Windows, a symlink elsewhere —
so the mirror of a large project stays a few megabytes.
Caveman Code: Reveal Shadow Workspace opens it; Delete Shadow Workspace removes it.
| Signal | Points |
|---|---|
| Produced a change at all | +100 |
| Build is clean | +400 |
| Each error the candidate introduced | −200 |
| Each existing error the candidate fixed | +150 |
| Each new warning | −20 |
| Each project-rule violation | −60 |
| Lines changed | −0.5 each, capped at −80 |
| Proposed nothing | −5000 |
Every candidate row in the UI can expand to show exactly which of these it earned.
.cavemanrules is normally prose the model may or may not honour. Forge compiles it into
constraints the scorer enforces, so the rules bite even when the model forgets them:
- Never use the `any` type in TypeScript -> forbid /�any�/ in **/*.{ts,tsx}
- Never use `console.log` in production code -> forbid /console\.log/
Only violations a candidate introduces are penalised, so an existing codebase full of them does not sink every proposal. For anything the heuristic cannot express, write the constraint directly:
forbid /TODO/ in src because ship finished code
require /^import/ in src because modules must import
Every write the agent makes snapshots the previous contents first, into the extension's
global storage (never your repo). Caveman Code: Rewind to Checkpoint lists them newest
first and restores one — including deleting files the agent created. Ctrl+Z does not
cover a multi-file change; this does.
Forge costs one model run per candidate plus a compile per unique candidate, so it is minutes, not seconds. Use it for changes that span files — renames, signature changes, refactors — where a single-shot local model is most likely to be confidently wrong. For questions and single-file tweaks, the normal chat path is faster and usually fine.
Set caveman.forge.autoForgeEdits to route every editing request through it.
Every claim about a local coding agent tends to rest on a demo that happened to go well. CavemanBench is the alternative: six realistic multi-file editing tasks, each with a programmatic verifier, run in a real temp project and judged by a real compiler.
npm run bench # every installed chat model
node scripts/bench.mjs qwen3.5:9b
node scripts/bench.mjs --repeat 3 --json results.jsonThere is also a Caveman Code: Benchmark My Models command. It scores your installed
models from inside the editor, opens the results as a markdown report, and offers to set
caveman.forge.models to the top performers — so the Forge ensemble is chosen from
measurements taken on your machine rather than a default picked on someone else's.
Each task targets a distinct way small models get multi-file edits wrong:
| Task | Catches |
|---|---|
rename-across-files |
Missing call sites in files it was never shown |
add-parameter |
Changing a signature without fixing callers |
create-and-wire |
Creating a file and never importing it |
fix-the-build |
Whether it can repair a broken build, not just avoid breaking one |
leave-it-alone |
Over-eagerness — editing correct code to look busy |
obey-project-rule |
Ignoring .cavemanrules when the easy answer violates it |
Passing needs both a clean compile and task-specific assertions: the rename task greps
for leftover references, create-and-wire checks the import actually exists, and
leave-it-alone fails a model that rewrites working code. A model cannot pass by luck,
and it cannot pass by doing nothing — except on the one task where doing nothing is right.
Temperature is pinned to 0 by default. A benchmark that moves under you is not a benchmark.
The harness is itself tested (src/test/bench.test.ts): scripted models with known-good
and known-bad answers are run through the real pipeline to prove the scoring is honest —
that a half-finished rename fails, that a forbidden any fails even though it compiles,
and that the fix-the-build fixture really does start broken.
The system prompt in src/prompt.ts was written by hand, which means every line in it is
a guess. scripts/prompt-evolve.mjs searches instead: each mutation is a hypothesis about
what makes a small model better at multi-file edits, and the benchmark decides which
hypotheses are true.
npm run evolve qwen3.5:9b
node scripts/prompt-evolve.mjs qwen3.5:9b --rounds 2The hypotheses currently under test:
| Mutation | Belief being tested |
|---|---|
plan-first |
Naming the files it will touch first reduces missed call sites |
repeat-one-file-rule |
Restating the one-block-per-file rule at the end beats stating it once |
search-before-rename |
An explicit "search first" rule for renames beats a general one |
terse-style |
The Style section is noise that costs attention |
no-worked-examples |
The examples are redundant next to the rules (expected to lose) |
It hill-climbs: evaluate the baseline and every mutation, adopt the winner, re-run the rest on top of it. Ties go to the incumbent — an unproven change is not an improvement, and a prompt that grows without evidence is how prompts rot.
| Command | Keybinding |
|---|---|
| Caveman Code: Open Chat | Ctrl+Shift+Alt+C |
| Caveman Code: Forge (verified edit) | Ctrl+Shift+Alt+F |
| Caveman Code: Edit Selection | Ctrl+Shift+Alt+E |
| Caveman Code: Ask About Selection | — |
| Caveman Code: Rewind to Checkpoint | — |
| Caveman Code: Benchmark My Models | — |
| Caveman Code: Select Model | — |
| Caveman Code: Reindex Workspace | — |
| Caveman Code: New Chat | — |
| Caveman Code: Reveal Shadow Workspace | — |
| Caveman Code: Delete Shadow Workspace | — |
| Caveman Code: Show Logs | — |
Type @path/to/file.ts in the composer to pin a specific file into context.
| Setting | Default | Notes |
|---|---|---|
caveman.endpoint |
http://127.0.0.1:11434 |
Ollama base URL |
caveman.model |
qwen3.5:9b |
Falls back to an installed model if missing |
caveman.temperature |
0.15 |
Low is better for code |
caveman.contextWindow |
8192 |
num_ctx; larger uses more VRAM |
caveman.maxIterations |
6 |
Agent tool-call rounds per request |
caveman.autoApplyEdits |
false |
Skip the review step |
caveman.commandApproval |
always |
always, safe (auto-run read-only commands), or never |
caveman.commandTimeoutSeconds |
120 |
Commands are killed after this |
caveman.maxContextFiles |
4 |
Files auto-attached per request |
caveman.maxFileBytes |
120000 |
Indexing/read size cap |
caveman.exclude |
node_modules, .git, build dirs, ... |
Glob patterns skipped by the indexer |
caveman.forge.candidates |
2 |
Ensemble size when forge.models is empty |
caveman.forge.models |
[] |
Explicit ensemble, e.g. ["qwen3.5:9b", "deepseek-r1:8b"] |
caveman.forge.repairRounds |
2 |
How many times the leader may repair from its own errors |
caveman.forge.includeTests |
false |
Also run the test suite as a referee (strongest, slowest) |
caveman.forge.autoForgeEdits |
false |
Route every editing request through Forge |
Shell commands always require approval unless commandApproval is safe and the
command is on the read-only allowlist. Destructive patterns (rm -rf, git push --force,
piping a download into a shell, ...) are flagged in the approval prompt and are never
auto-approved.
Create .cavemanrules in your workspace root:
## Code Style
- 4-space indentation
- Single quotes
- Always annotate return types
## Forbidden
- Never use the `any` type
It is appended to the system prompt on every request.
npm run compile # build
npm run watch # build on change
npm test # unit tests + webview harness (no GPU, no VS Code)
npm run test:ui # replays a session through webview/main.js on a fake DOM
npm run smoke # end-to-end protocol test against a real local model
npm run forge:demo # live Forge race against your installed models
node scripts/forge-trace.mjs <model> # print every turn one model takes
npm run bench # score models on CavemanBench
npm run evolve # search for a better system prompt
npm run lintnpm test covers the pure modules (parser, edit engine, retrieval scoring, command
safety, toolchain output parsing, rule compilation), a fake-DOM replay of a full
multi-turn session through the webview, and the entire Forge pipeline driven by a stub
model against a real tsc — including the two tests that encode the whole thesis:
✔ THE POINT: the compiler picks the correct model over the confident one
✔ THE OTHER POINT: a single weak model repairs itself from its own compiler errors
None of it needs VS Code or a GPU. npm run smoke sends
four real tasks to Ollama and checks that the model's output parses and applies — that is
the test that catches protocol regressions:
$ npm run smoke
Model: qwen3.5:9b
- edit an existing file... ok in 13.1s — +2/-2 lines, strategies: exact
- create a new file... ok in 3.3s — src/config.js, 2 lines
- answer a question without editing... ok in 4.6s — The server listens on port 3000.
- read a file, then edit it (2 turns)... ok in 5.5s — read then edited, strategies: exact
4/4 passed
Pass a model name to test a different one: node scripts/smoke.mjs deepseek-r1:8b.
src/
extension.ts activation, commands, status bar
chatView.ts webview host; agent + forge events <-> UI messages
agent.ts interactive agent: VS Code context -> a session
session.ts THE LOOP: stream a turn, run its actions, feed results back
forge.ts ensemble -> shadow -> toolchain -> score -> repair (pure, tested)
bench.ts CavemanBench tasks + runner + scoring (pure, tested)
forgeRunner.ts VS Code bridge for Forge: ensemble choice, staging
shadow.ts incremental project mirror with linked heavy dirs (pure, tested)
verify.ts toolchain detection + diagnostic parsing + deltas (pure, tested)
ruleCheck.ts .cavemanrules -> enforceable constraints (pure, tested)
checkpoint.ts time-travel snapshots (pure)
fileSource.ts read-only file access + copy-on-write overlay (pure, tested)
ollama.ts streaming NDJSON client for the Ollama HTTP API
protocol.ts parses model output into actions (pure, tested)
editEngine.ts SEARCH/REPLACE matching + line diffs (pure, tested)
scoring.ts keyword retrieval + symbol extraction (pure, tested)
commandRunner.ts shell execution + safety classification (pure, tested)
paths.ts globs, language detection (pure)
prompt.ts system/context prompt construction (pure)
indexer.ts workspace index and file tree
workspace.ts sandboxed file access
changeSet.ts staged changes, diff preview, apply/reject
webview/ chat UI + Forge race view (no dependencies, CSP-locked)
scripts/ bench, prompt evolution, forge demo, per-turn trace, UI harness
The `Overlay` is what lets one loop serve both paths: the interactive agent writes
through it to the real change set, while a Forge candidate writes through its own private
copy that never reaches disk.
There is no Python sidecar. Version 0.1 shelled out to a LangGraph process per prompt; that design was replaced, along with its cold-start cost and stdout-sentinel protocol, by the in-process TypeScript loop above.
MIT