Skip to content

About

Local-first AI coding agent for VS Code on Ollama. Ensemble candidates are compiled in a shadow workspace and your own toolchain picks the winner.

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

Caveman Code

A local-first AI coding agent for VS Code — where your compiler, not the model, decides what ships.

VS Code TypeScript Ollama Tests License


Caveman Code runs entirely against Ollama on your own machine — no cloud, no API keys, no telemetry.

Ask it questions about your codebase, or describe a change and review the diff before anything touches disk.

What it does

  • Chat side bar with streaming answers, rendered markdown, and collapsible model reasoning.
  • Agentic loop — the model can read files, search the workspace, list directories, and run shell commands (with your approval), then keep working from the results.
  • Reviewable edits — every change is staged as a diff card with Apply / Reject / Open diff. Nothing is written until you say so (unless you turn on caveman.autoApplyEdits).
  • Workspace index — file paths, languages and symbol names are indexed so the agent starts with relevant context instead of guessing.
  • Project rules — a .cavemanrules file in your workspace root is injected into every system prompt, and compiled into constraints the Forge scorer enforces.
  • Caveman Forge — several local models compete on the same task, each candidate is compiled and linted in a shadow copy of your project, and your toolchain picks the winner. The leader then repairs itself from its own compiler errors.
  • Time travel — every write is checkpointed, so any change is rewindable later.

Requirements

  • VS Code 1.85 or newer

  • Ollama running locally (ollama serve)

  • At least one chat model:

    ollama pull qwen3.5:9b

    Any instruction-following model works. Larger models follow the edit protocol more reliably; qwen3.5:9b is a good default on 8 GB+ of VRAM.

Getting started

npm install
npm run compile

Then press F5 in VS Code to launch the Extension Development Host, or package it:

npx vsce package
code --install-extension caveman-code-0.3.0.vsix

Open the Caveman icon in the activity bar, or press Ctrl+Shift+Alt+C.

How editing works

Small local models are bad at producing unified diffs — they get line numbers wrong almost every time. Caveman Code does not ask for one. Instead the model emits SEARCH/REPLACE blocks:

```caveman-edit src/server.ts
<<<<<<< SEARCH
const port = 3000;
=======
const port = Number(process.env.PORT ?? 3000);
>>>>>>> REPLACE
```

The SEARCH text is matched against the file by content, through a ladder of strategies:

  1. exact substring match
  2. line match ignoring trailing whitespace
  3. line match ignoring indentation, re-indenting the replacement to fit
  4. fuzzy window match, only above an 82% similarity threshold

A fuzzy match must also keep the block's identifiers: export function subtract(a) is over 90% character-similar to export function add(a), so similarity alone would happily rewrite the wrong function. At least 80% of the identifiers in the SEARCH text have to be present in the matched window, which is what stops that.

The resulting file content is computed locally and applied as a single whole-document write, so there is no line arithmetic to get wrong.

When a block does not apply

Small models reliably make one specific mistake: they put SEARCH/REPLACE pairs for several files into a single block addressed to the first one. Rather than rejecting the whole block:

  • the hunks that did match are still applied, so the model does not have to redo them;
  • each hunk that failed is looked up across the workspace, and if its text lives in another file the model is told exactly which one — "Change 2 is not in src/math.ts, that text is in src/totals.ts. Put it in its own block."
  • only genuinely unlocatable hunks fall back to "here is the closest text in the file".

Before this, a model that made that mistake would loop on it until it ran out of steps. After it, the same model finishes the same task in three turns. scripts/forge-trace.mjs prints every turn a model takes and is how that gets diagnosed:

$ node scripts/forge-trace.mjs qwen3.5:9b
── turn 1 ──  actions: Search "add"
── turn 2 ──  actions: Edit src/math.ts | Edit src/totals.ts | Edit src/report.ts ...
── turn 3 ──  actions: (none)   >>> no actions: the loop ends here
turns: 3   edits: 3 file(s)   elapsed: 23.6s

The same fenced-block protocol covers the other actions:

Block Purpose
caveman-read <path> Read a file into the conversation
caveman-search Search the workspace (body is the query; /regex/ supported)
caveman-list <dir> List a directory
caveman-edit <path> SEARCH/REPLACE edit
caveman-create <path> New file, body is the full contents
caveman-delete <path> Delete a file (moved to trash)
caveman-run Shell command, subject to approval

Caveman Forge — verification-driven editing

A 9B model cannot reliably one-shot a correct edit. Forge stops treating the model as an oracle and starts treating it as a search process, with your own toolchain as the referee.

Press the Forge button in the composer (or Ctrl+Shift+Alt+F) and one request becomes:

  1. An ensemble proposes. Several local models answer the same question independently. Different architectures make different mistakes.

  2. Each candidate lands in a shadow workspace — a real mirror of your project in the temp directory. Your files are never touched.

  3. Your real toolchain judges it. Whatever the project already uses: tsc, ESLint, cargo check, go build, Python syntax, and optionally your test suite. Nothing is installed or downloaded; only tools already present are run.

  4. Candidates are scored on what they broke, what they fixed, whether they obey .cavemanrules, and how small the change is.

  5. The leader repairs itself. The winning candidate is handed its own compiler errors and tries again — and files named in those errors are pulled into its context automatically. The compiler turns out to be a far better retrieval signal than keyword search.

    Three guarantees hold here: a repair that scores worse than the state it started from is rolled back (candidate, overlay and conversation together); a round that answers without editing anything is retried once, more forcefully, rather than burning the budget; and a candidate is never left mid-phase, so nothing spins forever in the race view.

  6. Only the winner is shown to you, as a normal reviewable diff — and if even the best candidate still does not compile, the UI says so above the diff in red, listing the remaining errors. Forge offers the best candidate it found, which is not always a working one, and a diff that looks reviewed must never imply it was verified.

Here is a real run against a local ensemble, on the classic failure mode — a rename whose call sites live in files the model was never shown:

$ node scripts/forge-demo.mjs qwen3.5:9b

  · referees: TypeScript
  · baseline: 0 error(s), 0 warning(s) in 1566ms
  · qwen3.5:9b: score -104 (1 new problems, 0 fixed, +4/-4)
  · qwen3.5:9b: after repair 1, score 495 (0 new problems)

  CANDIDATE            VERDICT                            CHURN      SCORE   TIME
  ------------------------------------------------------------------------------
  qwen3.5:9b           compiles clean [repaired x1]       +5/-5      495     29.1s

  WINNER: qwen3.5:9b — toolchain clean
  files:  src/math.ts, src/totals.ts, src/report.ts
  source: UNCHANGED on disk (nothing applied without review)

The model made exactly the mistake a 9B model makes — it renamed the definition and missed a caller in a file it had never been shown. tsc caught it, the repair round fixed it, and what reached the review UI compiled. One 9B model, 29 seconds, verified.

That single-model run is the sharper demonstration: the ensemble helps, but the loop is what turns an unreliable model into a reliable one.

The shadow workspace

Candidates are compiled somewhere else entirely, because verification means running real tools that want a real directory tree. Editing your files and undoing them would fight your editor, your file watchers and your git status — and leave you broken if anything crashed mid-verify.

The mirror is persistent and synced incrementally: the first sync copies your source files, every later one is a stat walk. Heavy directories (node_modules, target, .venv) are linked, not copied — a directory junction on Windows, a symlink elsewhere — so the mirror of a large project stays a few megabytes.

Caveman Code: Reveal Shadow Workspace opens it; Delete Shadow Workspace removes it.

Scoring

Signal Points
Produced a change at all +100
Build is clean +400
Each error the candidate introduced −200
Each existing error the candidate fixed +150
Each new warning −20
Each project-rule violation −60
Lines changed −0.5 each, capped at −80
Proposed nothing −5000

Every candidate row in the UI can expand to show exactly which of these it earned.

Executable project rules

.cavemanrules is normally prose the model may or may not honour. Forge compiles it into constraints the scorer enforces, so the rules bite even when the model forgets them:

- Never use the `any` type in TypeScript      ->  forbid /�any�/ in **/*.{ts,tsx}
- Never use `console.log` in production code  ->  forbid /console\.log/

Only violations a candidate introduces are penalised, so an existing codebase full of them does not sink every proposal. For anything the heuristic cannot express, write the constraint directly:

forbid /TODO/ in src because ship finished code
require /^import/ in src because modules must import

Time travel

Every write the agent makes snapshots the previous contents first, into the extension's global storage (never your repo). Caveman Code: Rewind to Checkpoint lists them newest first and restores one — including deleting files the agent created. Ctrl+Z does not cover a multi-file change; this does.

When to use it

Forge costs one model run per candidate plus a compile per unique candidate, so it is minutes, not seconds. Use it for changes that span files — renames, signature changes, refactors — where a single-shot local model is most likely to be confidently wrong. For questions and single-file tweaks, the normal chat path is faster and usually fine.

Set caveman.forge.autoForgeEdits to route every editing request through it.

CavemanBench — measuring instead of guessing

Every claim about a local coding agent tends to rest on a demo that happened to go well. CavemanBench is the alternative: six realistic multi-file editing tasks, each with a programmatic verifier, run in a real temp project and judged by a real compiler.

npm run bench                    # every installed chat model
node scripts/bench.mjs qwen3.5:9b
node scripts/bench.mjs --repeat 3 --json results.json

There is also a Caveman Code: Benchmark My Models command. It scores your installed models from inside the editor, opens the results as a markdown report, and offers to set caveman.forge.models to the top performers — so the Forge ensemble is chosen from measurements taken on your machine rather than a default picked on someone else's.

Each task targets a distinct way small models get multi-file edits wrong:

Task Catches
rename-across-files Missing call sites in files it was never shown
add-parameter Changing a signature without fixing callers
create-and-wire Creating a file and never importing it
fix-the-build Whether it can repair a broken build, not just avoid breaking one
leave-it-alone Over-eagerness — editing correct code to look busy
obey-project-rule Ignoring .cavemanrules when the easy answer violates it

Passing needs both a clean compile and task-specific assertions: the rename task greps for leftover references, create-and-wire checks the import actually exists, and leave-it-alone fails a model that rewrites working code. A model cannot pass by luck, and it cannot pass by doing nothing — except on the one task where doing nothing is right.

Temperature is pinned to 0 by default. A benchmark that moves under you is not a benchmark.

The harness is itself tested (src/test/bench.test.ts): scripted models with known-good and known-bad answers are run through the real pipeline to prove the scoring is honest — that a half-finished rename fails, that a forbidden any fails even though it compiles, and that the fix-the-build fixture really does start broken.

Prompt evolution — an executable fitness function

The system prompt in src/prompt.ts was written by hand, which means every line in it is a guess. scripts/prompt-evolve.mjs searches instead: each mutation is a hypothesis about what makes a small model better at multi-file edits, and the benchmark decides which hypotheses are true.

npm run evolve qwen3.5:9b
node scripts/prompt-evolve.mjs qwen3.5:9b --rounds 2

The hypotheses currently under test:

Mutation Belief being tested
plan-first Naming the files it will touch first reduces missed call sites
repeat-one-file-rule Restating the one-block-per-file rule at the end beats stating it once
search-before-rename An explicit "search first" rule for renames beats a general one
terse-style The Style section is noise that costs attention
no-worked-examples The examples are redundant next to the rules (expected to lose)

It hill-climbs: evaluate the baseline and every mutation, adopt the winner, re-run the rest on top of it. Ties go to the incumbent — an unproven change is not an improvement, and a prompt that grows without evidence is how prompts rot.

Commands

Command Keybinding
Caveman Code: Open Chat Ctrl+Shift+Alt+C
Caveman Code: Forge (verified edit) Ctrl+Shift+Alt+F
Caveman Code: Edit Selection Ctrl+Shift+Alt+E
Caveman Code: Ask About Selection —
Caveman Code: Rewind to Checkpoint —
Caveman Code: Benchmark My Models —
Caveman Code: Select Model —
Caveman Code: Reindex Workspace —
Caveman Code: New Chat —
Caveman Code: Reveal Shadow Workspace —
Caveman Code: Delete Shadow Workspace —
Caveman Code: Show Logs —

Type @path/to/file.ts in the composer to pin a specific file into context.

Settings

Setting Default Notes
caveman.endpoint http://127.0.0.1:11434 Ollama base URL
caveman.model qwen3.5:9b Falls back to an installed model if missing
caveman.temperature 0.15 Low is better for code
caveman.contextWindow 8192 num_ctx; larger uses more VRAM
caveman.maxIterations 6 Agent tool-call rounds per request
caveman.autoApplyEdits false Skip the review step
caveman.commandApproval always always, safe (auto-run read-only commands), or never
caveman.commandTimeoutSeconds 120 Commands are killed after this
caveman.maxContextFiles 4 Files auto-attached per request
caveman.maxFileBytes 120000 Indexing/read size cap
caveman.exclude node_modules, .git, build dirs, ... Glob patterns skipped by the indexer
caveman.forge.candidates 2 Ensemble size when forge.models is empty
caveman.forge.models [] Explicit ensemble, e.g. ["qwen3.5:9b", "deepseek-r1:8b"]
caveman.forge.repairRounds 2 How many times the leader may repair from its own errors
caveman.forge.includeTests false Also run the test suite as a referee (strongest, slowest)
caveman.forge.autoForgeEdits false Route every editing request through Forge

Shell commands always require approval unless commandApproval is safe and the command is on the read-only allowlist. Destructive patterns (rm -rf, git push --force, piping a download into a shell, ...) are flagged in the approval prompt and are never auto-approved.

Project rules

Create .cavemanrules in your workspace root:

## Code Style
- 4-space indentation
- Single quotes
- Always annotate return types

## Forbidden
- Never use the `any` type

It is appended to the system prompt on every request.

Development

npm run compile      # build
npm run watch        # build on change
npm test             # unit tests + webview harness (no GPU, no VS Code)
npm run test:ui      # replays a session through webview/main.js on a fake DOM
npm run smoke        # end-to-end protocol test against a real local model
npm run forge:demo   # live Forge race against your installed models
node scripts/forge-trace.mjs <model>   # print every turn one model takes
npm run bench        # score models on CavemanBench
npm run evolve       # search for a better system prompt
npm run lint

npm test covers the pure modules (parser, edit engine, retrieval scoring, command safety, toolchain output parsing, rule compilation), a fake-DOM replay of a full multi-turn session through the webview, and the entire Forge pipeline driven by a stub model against a real tsc — including the two tests that encode the whole thesis:

✔ THE POINT: the compiler picks the correct model over the confident one
✔ THE OTHER POINT: a single weak model repairs itself from its own compiler errors

None of it needs VS Code or a GPU. npm run smoke sends four real tasks to Ollama and checks that the model's output parses and applies — that is the test that catches protocol regressions:

$ npm run smoke
Model:    qwen3.5:9b

- edit an existing file... ok in 13.1s — +2/-2 lines, strategies: exact
- create a new file... ok in 3.3s — src/config.js, 2 lines
- answer a question without editing... ok in 4.6s — The server listens on port 3000.
- read a file, then edit it (2 turns)... ok in 5.5s — read then edited, strategies: exact

4/4 passed

Pass a model name to test a different one: node scripts/smoke.mjs deepseek-r1:8b.

Architecture

src/
  extension.ts     activation, commands, status bar
  chatView.ts      webview host; agent + forge events <-> UI messages
  agent.ts         interactive agent: VS Code context -> a session
  session.ts       THE LOOP: stream a turn, run its actions, feed results back
  forge.ts         ensemble -> shadow -> toolchain -> score -> repair  (pure, tested)
  bench.ts         CavemanBench tasks + runner + scoring                (pure, tested)
  forgeRunner.ts   VS Code bridge for Forge: ensemble choice, staging
  shadow.ts        incremental project mirror with linked heavy dirs    (pure, tested)
  verify.ts        toolchain detection + diagnostic parsing + deltas    (pure, tested)
  ruleCheck.ts     .cavemanrules -> enforceable constraints             (pure, tested)
  checkpoint.ts    time-travel snapshots                                (pure)
  fileSource.ts    read-only file access + copy-on-write overlay        (pure, tested)
  ollama.ts        streaming NDJSON client for the Ollama HTTP API
  protocol.ts      parses model output into actions                     (pure, tested)
  editEngine.ts    SEARCH/REPLACE matching + line diffs                 (pure, tested)
  scoring.ts       keyword retrieval + symbol extraction                (pure, tested)
  commandRunner.ts shell execution + safety classification              (pure, tested)
  paths.ts         globs, language detection                            (pure)
  prompt.ts        system/context prompt construction                   (pure)
  indexer.ts       workspace index and file tree
  workspace.ts     sandboxed file access
  changeSet.ts     staged changes, diff preview, apply/reject
webview/           chat UI + Forge race view (no dependencies, CSP-locked)
scripts/           bench, prompt evolution, forge demo, per-turn trace, UI harness

The `Overlay` is what lets one loop serve both paths: the interactive agent writes
through it to the real change set, while a Forge candidate writes through its own private
copy that never reaches disk.

There is no Python sidecar. Version 0.1 shelled out to a LangGraph process per prompt; that design was replaced, along with its cold-start cost and stdout-sentinel protocol, by the in-process TypeScript loop above.

License

MIT

About

Local-first AI coding agent for VS Code on Ollama. Ensemble candidates are compiled in a shadow workspace and your own toolchain picks the winner.

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages