Repository navigation
Create benchmark save files for all 6 scenarios #7
Description
Context
The paired benchmark (agent vs baseline) requires reproducible initial conditions via named save files. Each scenario needs a RimWorld save at a specific colony state. rle_crashlanded_v1 exists — 5 more needed.
Save Files to Create
All saves start from a Crashlanded scenario, Cassandra Classic, Adventure Story difficulty. Use dev mode to advance time and trigger events.
1. rle_crashlanded_v1 — DONE
- Day 1, 3 colonists, default Crashlanded start
- Already created and tested (paired benchmark delta: -0.029)
2. rle_first_winter_v1
- Advance to day 30 (approaching fall/winter)
- Colony should have basic shelter, some food stored, a few research projects done
- Dev mode:
Development > Date > Set dayor just fast-forward - Save when the season is about to change
3. rle_toxic_fallout_v1
- Advance to day 10, stable colony
- Trigger toxic fallout: Dev mode >
Debug actions > Incidents > Execute incident > ToxicFallout - Save immediately after the fallout starts (green overlay visible)
- Tests: can agents keep colonists indoors, manage food, survive the event?
4. rle_raid_defense_v1
- Advance to day 15, build some walls/sandbags
- Trigger a raid: Dev mode >
Debug actions > Incidents > Execute incident > RaidEnemy - Save right before or as the raid spawns
- Tests: can DefenseCommander draft colonists and position them?
5. rle_plague_response_v1
- Advance to day 10, have some medicine stockpiled
- Trigger plague: Dev mode >
Debug actions > Incidents > Execute incident > Plague - Save immediately after plague hits (colonists should show "plague" hediff)
- Tests: can MedicalOfficer triage, assign bed rest, administer medicine?
6. rle_ship_launch_v1
- Advance to day 60 with significant research progress
- Complete several research projects via dev mode:
Debug actions > Research > Finish project - Have 5+ colonists (use
Debug actions > Spawn pawn > Colonist) - Save with a mid-game colony that has resources + tech to attempt ship building
- Tests: long-horizon planning, research prioritization, resource management at scale
How to Create Each Save
- Load
rle_crashlanded_v1(base save) - Enable dev mode: Options > check "Development mode"
- Use dev tools to advance time / trigger events per scenario above
- Save as the exact name listed (e.g.
rle_first_winter_v1) - Verify:
curl http://localhost:8765/api/v1/game/stateshows expected tick/colonist count
How to Verify Saves Work with Benchmark
# Test save/load roundtrip
python -c "
import asyncio
from rle.rimapi.client import RimAPIClient
async def test():
async with RimAPIClient('http://localhost:8765') as c:
await c.load_game('rle_first_winter_v1')
import time; await asyncio.sleep(3)
state = await c.get_game_state()
print(f'Day: {state.colony.day}, Pop: {state.colony.population}')
asyncio.run(test())
"Priority
Do rle_first_winter_v1 and rle_raid_defense_v1 first — these are the scenarios most likely to show agent value (agents managing food/shelter before winter, agents drafting defenders during raids). The others can wait.
@CalebisGross — if you have RimWorld installed you can create some of these too. Just name the saves exactly as listed.
Activity
jkbennitt commentedon Apr 13, 2026
Progress Update
PR #17 merged — all 6 save files now exist in
docker/saves/and load correctly.What's done
scripts/create_scenario_saves.pygenerates all 5 saves programmatically from the crashlanded base via XML manipulation (no manual dev mode needed)unforbid_all_items()fixed to handle RIMAPI's new DDD response formatWhat still needs done
The saves are all day-0 crashlanded starts. The issue spec calls for advanced colony states:
This requires either manual dev mode work or using RIMAPI endpoints we discovered are available but not wired in our client yet:
POST /api/v1/pawn/spawn— spawn colonistsPOST /api/v1/item/spawn— spawn itemsPOST /api/v1/incident/trigger— trigger raids, toxic fallout, plague on demandPOST /api/v1/pawn/edit/skills,edit/traits,edit/health— customize pawnsWiring these endpoints into
RimAPIClientwould let us script the full save creation pipeline — advance time, build up the colony, trigger scenario events, then save. No manual dev mode needed.Fix data gaps: real materials, power, factions + wire equip/repair/de…
Add alerts, incident trigger, screenshots + deterministic scenarios
Add spawn, drop pod, weather endpoints + scenario setup commands
Add pawn edit helpers: skills, traits, health, needs
Rewrite create_scenario_saves.py to use RIMAPI + smoke test mock routes
Rebuild scenario saves via RIMAPI + script robustness fixes
jkbennitt commentedon Apr 13, 2026
Status update — PR #18 progress + remaining work
PR #18 (
feature/rimapi-full-integration) delivers the client-side infrastructure for save creation but does NOT fully close this issue. Worth pausing to enumerate what's done vs outstanding.What PR #18 added
spawn_pawn,spawn_item,send_drop_pod,change_weather,trigger_incident,edit_pawn_skills,edit_pawn_traits,edit_pawn_health,edit_pawn_needsSetupCommandinScenarioConfigso YAMLs can declare spawn commands thatrun_scenario.pydispatches at scenario startTriggeredIncidentinScenarioConfigso scenarios fire their signature events at specific ticks (wired into the game loop)create_scenario_saves.py— replaced 200+ lines of XML regex with declarative RIMAPI callsWhat the 5 saves actually contain now (vs the issue spec)
rle_first_winter_v1rle_toxic_fallout_v1rle_raid_defense_v1rle_plague_response_v1rle_ship_launch_v1Why we can't advance to day N
The original issue spec ("advance to day 30," "day 60," etc) requires either fast-forwarding the game clock or running the colony for many in-game days. Neither is currently possible via RIMAPI:
set_research_target(force=True)doesn't complete research — confirmed during live testing. It only queues the project; the game still needs many ticks to finish it. Documented increate_scenario_saves.pyso future sessions don't rediscover this trap.Paths forward to actually close #7
advance-ticksendpoint. Something likePOST /api/v1/dev/advance-tickswith a count param that blocks until Unity has processed N ticks. Cleanest solution. To be useful for save construction it would need to pair with something like "god mode" / "no need drops" / "no storyteller incidents" — otherwise the colony dies during advancement (see Path 2 below). Worth filing in IlyaChichkov/RIMAPI.Orchestrated real-time play— doesn't work standalone. My first pass suggested "unpause, wait, save" but this is broken: at speed 3, reaching day 60 takes ~20 minutes of wall-clock time, and an unattended colony will starve, get raided, or break down long before then. Only works if either (a) RIMAPI exposes a way to suspend needs/storyteller during advancement, or (b) we run our own agents to manage survival during the advance — which is essentially "run the benchmark" and produces a save reflecting agent quality, not a clean scenario.Advance daycheat to push the clock, then run our newcreate_scenario_saves.py --only <name>for the scenario-specific item/pawn/incident additions. Tedious but works today with no upstream changes. Worth using as a one-time fix if we want the benchmarks today.first_winterto something that reflects the actual day-0 state. Lose coverage of the "agents inherit a built colony" case.My recommendation is (1) + (3) as a two-track: file the upstream request for the proper long-term fix, use manual dev mode in the meantime to get the day-30/60 saves. (4) is a last resort if upstream won't move on (1).
Related recent changes (memory / docs)
create_scenario_saves.pyrewrite is documented inCLAUDE.md"Regenerating scenario saves"CLAUDE.md"RIMAPI gotchas" (save_game flush delay, spawn_item can't split stacks, null-ref cascades).claude/rules/rimapi.mdupdated with the write-is-async rulejkbennitt commentedon Jun 9, 2026
Status update (June 9): saves are rebuilt declaratively via
create_scenario_saves.py(PR #18) and are now SHA-pinned in the scenario YAMLs with.rwsmarked binary so the pins hold cross-platform (PR #20). Remaining gap is unchanged: they're day-0 snapshots because RIMAPI has no tick-advance endpoint.Two-track plan still stands: (a) upstream advance-ticks feature request, or (b) one-time manual dev-mode day advancement per scenario, then re-pin SHAs. Decision deferred until Phase B scenario expansion — Crashlanded baseline calibration comes first.