TreasureMap

Human draws the map. AI runs the expedition.

TreasureMap is a small framework for map-driven autonomous research. A human writes a research map: the problem, what counts as treasure, the dataset, model constraints, evaluation rules, and a list of ideas. The system turns that map into a JackMap-style expedition: it builds a harness, checks which ideas are runnable, executes bounded iterations, and leaves an auditable trail.

The point is not to let an agent wander freely. The point is to give it a map, a harness, and a narrow place to dig.

Motivation

TreasureMap is inspired by Andrej Karpathy's AutoResearch framing: if you already have a pile of research ideas, the bottleneck is often not having ideas. The bottleneck is turning them into small tests, building the harness, running the boring comparisons, and figuring out which thread looks interesting enough to pull next.

That feels like exactly the kind of work an AI agent should help with.

The human still provides taste and direction:

• here is the problem
• here are my guesses
• here is the dataset or environment
• here is what would count as treasure
• here are the boundaries

Then the AI does the expedition work:

• build or repair the harness
• preflight each idea
• run bounded experiments
• keep the evidence trail
• report which ideas look promising, boring, broken, or blocked

The Loop

human map
  -> bootstrap harness
  -> validate harness
  -> preflight ideas
  -> run bounded iterations
  -> write report

Each idea is classified before execution:

• ready_exact: the harness can test it directly
• ready_proxy: the harness can only run a declared proxy
• needs_harness: more harness work is needed first
• blocked: the idea is outside the current safe scope

Core Idea

Most AI research automation demos collapse into one big prompt. TreasureMap keeps the work split:

• Map: human intent, hypotheses, boundaries, and success signals
• Harness: executable world with data, metrics, adapters, and validation
• Preflight: the gate that decides whether an idea can run
• Iteration: the AI's bounded attempt
• Report: evidence separated from speculation

For ML research, the flagship pattern is the constrained experiment slot. After the harness owns data loading, train/test split, metrics, and artifacts, the AI may only edit one file:

harnesses/<map_slug>/experiment.py

The candidate is accepted only if the harness passes and the locked files remain unchanged.

There is one earlier repair phase: if the generated harness itself does not pass validation, Codex can repair only the declared harness files. Once validation passes, the harness is locked and the experiment worker is narrowed to experiment.py.

Quick Start

cd treasuremap
python3 -m pip install -e ".[dev]"
cp .env.example .env

Put your OpenAI key in .env:

OPENAI_API_KEY=...
TREASUREMAP_MODEL=gpt-5.4

Run tests:

python3 -m pytest -q

Run the real ML AutoResearch demo:

python3 -m treasuremap jackmap-expedition \
  --map maps/research_ml_wine_autoresearch.md \
  --executor openai \
  --model gpt-5.4 \
  --iterations 1

Demo: Real ML AutoResearch

The recommended demo map is:

maps/research_ml_wine_autoresearch.md

It uses the UCI red wine quality dataset and asks whether an AI experiment worker can improve a held-out predictor while the harness keeps the research surface fixed.

The harness owns:

• public dataset caching
• deterministic train/test split
• regression metrics: RMSE, MAE, R2
• artifact writing
• locked-file verification

Before the harness is locked, Codex can repair only:

run_harness.py
experiment.py
manifest.json
dataset.jsonl
README.md

After the harness passes validation, the AI experiment worker controls only:

harnesses/wine_quality_autoresearch_ml/experiment.py

In the latest local run, GPT-5.4 produced a constrained experiment.py replacement that improved held-out RMSE from the mean baseline to a kNN-style regressor. The generated run artifacts stay local by default and are not committed.

Map Format

Maps live in maps/*.md:

# My Research Map

## Problem

What are we trying to learn?

## Treasure

What would count as useful evidence?

## Dataset

What data should the harness use or create?

## Model

What modeling constraints should the harness respect?

## Evaluation

What metrics or checks decide whether a run is useful?

## Ideas

### 1. First idea
Capability: some_capability
Priority: high
Success: A concrete success signal.

Commands

Build a harness:

python3 -m treasuremap bootstrap \
  --map maps/research_ml_wine_autoresearch.md \
  --apply-manifest

Run the JackMap-style expedition:

python3 -m treasuremap jackmap-expedition \
  --map maps/research_ml_wine_autoresearch.md \
  --executor openai \
  --model gpt-5.4 \
  --iterations 1

Refresh a report:

python3 -m treasuremap report --map maps/research_ml_wine_autoresearch.md

Run a dry protocol check without OpenAI:

python3 -m treasuremap jackmap-expedition \
  --map maps/research_ml_wine_autoresearch.md \
  --executor deterministic \
  --iterations 1

Repository Layout

• maps/: human-written treasure maps
• rules/: global execution and reporting rules
• src/treasuremap/: framework code
• tests/: parser, harness, and runner tests
• harnesses/: generated harnesses, ignored by default
• runs/: generated iteration records, ignored by default
• reports/: generated reports, ignored by default

Public Repo Safety

Never commit .env or raw API responses. This repo ignores .env, harnesses/, runs/, reports/, Python caches, and raw OpenAI output by default.

If you have ever pasted a real API key into chat or shell history, rotate it before publishing.

License

MIT

Status

Early research scaffold. The current best path through the project is the ML AutoResearch map:

maps/research_ml_wine_autoresearch.md

It demonstrates the core promise: humans provide the research map; the system builds the harness; the AI digs inside a constrained experiment slot; the artifacts show what happened.