Repository navigation
feat: experiment progress visualization and watching - #1031
Draft
small-angel wants to merge 58 commits into
Draft
small-angel wants to merge 58 commits into
small-angel wants to merge 58 commits into
Conversation
Contributor
Author
|
(claude is still fixing something before it can push my bad) |
Draft: the run-monitoring stack as one commit for trying out. Agents instrument an experiment's stages and measures and write its view; runs show progress, time left and their charts above the chat and on the page, several runs share one summary, and an uninstrumented run says it is setting up and reads progress from its log. Assisted-by: claude-fable-5.2 Assisted-by: claude-opus-5.5 Assisted-by: claude-sonnet-5.5 Assisted-by: gpt-6.1-sol
small-angel
force-pushed
the
clo/run-monitoring
branch
from
November 14, 2026 02:34
664ecf3 to
b743eb0
Compare
A reusable test bed for the rules, tripwires and watcher that are about to land: an invented training run (payload.mjs) in twelve variants, each a child of a healthy parent, and a runner that launches one against the real orx up in the e2e sandbox and prints the line, strip, page, status text, monitor flag/sentence/watch fields and the wake message the agent gets at fixed seconds after the child starts. Scenarios live beside the payload so a dev project can use the same run command. The runner counts points from the run's startedAt, since a busy machine can take most of a minute to submit one. Assisted-by: claude-sonnet-5.5
An agent knows a run's whole lifetime before it starts, so the whole-run
bar no longer counts only the stages seen so far ('50%' during download,
'100% · 0s left' at the end of train with eval and SFT to come).
- Stored: the experiment's monitor settings gain plan: [{phase, of?,
unit?}], set with 'orx exp view set EXP --config PATH' ({"plan": …};
any other key is refused, naming what this orx reads), shown by 'view
get', copied into children with the rest of the settings. An empty plan
clears it. A stage that runs again is listed again where it recurs,
never twice in a row, since a log cannot tell two such occurrences
apart.
- Wire: '::orx:: {"plan": […]}' is the run's whole plan and replaces the
experiment's for that run; keys a stage does not know are ignored. The
checkpoint keeps it as 'plan'. orx_run.plan([...]) writes it.
- Stages: occurrences are laid over the plan in order. A stage the plan
does not name is shown as unplanned and the plan grows by it; planned
stages passed over, or never reached by a run that finished, are
skipped. Each stage's weight is how long the parent's run took over the
stage of the same name and count. When any stage has no such time, the
whole-run fraction is unknown (the bar is drawn as steps), never an
equal split; time left adds the parent's later stages and is a floor
('at least', naming the untimed stages) when one is unknown.
- Surfaces: the monitor lists every stage with planned/skipped,
segments span all of them, the summary carries segments for the line,
and 'orx exp status' prints a stages line. A run with no plan is
projected as before.
Assisted-by: claude-opus-5.5
The strip and the page list a plan's stages as the prototype does: done
stages with their time, the current one, the ones to come with 'about
Xm' from the parent's run, a skipped stage marked skipped (never done)
and an unplanned one marked unplanned. The strip lists them under its
chart now, not only when the view has no block for the stage.
The line's underline is segmented by stage from the summary's new
segments, so it needs no full monitor. A time left that leaves out
untimed stages names them ('No earlier run timed eval'); the summary
carries etaUntimed so the line and the strip say the same, and
'orx exp status' uses the same words. A stage that has begun with no
step yet reads as its name in the caption.
Assisted-by: claude-opus-5.5
'orx exp view get' printed a declared total of 3 shards as 3.0; the stored plan keeps the number the agent wrote. Assisted-by: claude-opus-5.5
A finished run whose last stage reported no step (chat) read no share at all, and with priors could read under 100%. Every stage of a plan that finished is done; the stages it skipped are not segments. Assisted-by: claude-opus-5.5
An agent picks, per view, what a run is compared with and what of the parent is drawn, since that depends on the experiment at hand: - vs="step|final" on <stats> and on a <signal> as its default, and vs="parent|step|final" on <table>: at the same stage and step, or with where the parent ended. Without one, a run compares at the same step while it goes and with the parent's final value once it ended. A stage that has ended compares at its last step, so a run in eval no longer measures its train loss against the parent's final one. - ghost="true|false" on <series>: a reference the run measures itself is not drawn twice. Every line of a chart with ghost="parent" has the parent's under it by default, over steps too. - x-range="run|parent" on a chart over steps: by default the axis spans the run's own total and the parent's line is cut at the edge, so a short child is no longer squashed under its parent's 5,000 steps, and the line no longer runs past a strip's edge. The monitor gains versusBy (both ways, for a block that names its own) and a step on each comparison made at one; the status line says which. Assisted-by: claude-opus-5.5
An agent asked mid-run to show something else runs `orx exp view set` with a new view and new patterns; the open strip and page redraw with no reload, and the patterns read what the log already said before they existed. The test records when each surface first shows the change: the page within the command's own return, the strip's new headline with its value at the next collection, about 0.8 s later. Assisted-by: claude-opus-5.5
The view spec read the base run's line after the agent had said what came of the run, when the line may already have left; it failed about half the full-suite runs. The result turn now holds until the line is read. Assisted-by: claude-opus-5.5
`versus` held each signal compared its own way and `versusBy` held it
both ways, so every own-way entry went out twice. Nothing has shipped,
so `versus` now carries the two ways, `{"step": …, "final": …}`, and
`versusBy` is gone without a compatibility path. The page picks the way
a block names, else the signal's own `vs`, else the same step while
the run goes and where the parent ended once it ended (`versusOf`); the
server keeps the same rule for the summary, the bindings and the status
text.
Assisted-by: claude-opus-5.5
A time left that is only a floor ("at least", from a counter in the log
or a plan with stages no earlier run timed) counted down to "At least 0s
left", which promises nothing. Under a second left it now reads "No
estimate", with the stages no earlier run timed as the detail ("No
earlier run timed eval, sft") or, for a counter read from the log, that
more work may follow it. `orx exp status` says the same. A plan whose
known stages are done and whose rest is untimed keeps a zero floor
instead of no estimate, so it names those stages too.
Assisted-by: claude-opus-5.5
How long a stage should take now comes from the experiment's own last run that finished, then from the parent's, stage by stage (PLAN §4 order). Before, only a run with a declared plan had stage times at all, and only from its parent. A run without a plan now gets them for every stage it reaches, so a run that goes silent at its first step stalls against the earlier pace (the `hang` fault: "no progress" after 90 s where it said "Estimating…" forever), and its first slices blend the earlier time as the estimator says. Assisted-by: claude-opus-5.5
A run that declared no plan and had reached a second stage read a share of the whole run from an equal split across the stages seen so far: 50% the moment train began, 98% just before an eval no one had seen yet. The stages still to come are unknown, so the share is too: the bar keeps its stages and the caption drops the percentage, as a plan with an untimed stage already does. Assisted-by: claude-opus-5.5
The line led with the first of the view's headline signals that held a number, so a finished run hid a NaN loss behind its eval score (`nan-400`) and an all-zero eval behind a level val loss (`eval-zeros`). The rule now: among the view's headline and then its other signals, the first non-finite value leads; else the first that is exactly zero where the parent's is not; otherwise the first headline with a number, as before. The server decides and sends the name as `headline` on the monitor and in its events, and the page draws it instead of choosing again. Assisted-by: claude-opus-5.5
A Python script with no shell around it could not name the stage it starts in: `orx_run.progress()` wrote its first report before any phase, so the stage list began with an unnamed stage the plan never listed. `progress(phase="train", …)` puts the name in that first report. Assisted-by: claude-opus-5.5
`orx exp status` and the line's summary said "eval · step 10" when the run printed steps without a total, while the stage list beside it had the plan's total. Both now take the plan's total when the run printed none: "eval · step 10 of 50". Assisted-by: claude-opus-5.5
Past its estimate with no progress since, a run's time left sat at "0s left" for as long as it stayed silent, beside a bar that had stopped (`hang-mid`, `step-stuck`: 40 to 80 s before the stall note, and then only in the strip). Once the countdown runs out with no progress since, the time slot reads how long it has been silent, "No progress for 40s", with "was due about 7:41 PM" as the detail, on the line, the strip and in `orx exp status`. A stage that had nothing left when estimated keeps "0s left"; a floor keeps its own words. Assisted-by: claude-opus-5.5
A run that failed without a message read "no message recorded" in `orx exp status`, "exit 1" on its page and nothing in its strip, though its log ended in a traceback that only the agent's wake carried. The log's last error now says it: a Python traceback's exception line, else the last line that names an error. `orx exp status` reads "reason: last error in the log: `torch.OutOfMemoryError: …`", the wake's Reason says the same, and the strip and page quote it under the Failed lead. A line the credential scan flags is never shown; a log that cannot be read leaves the reason pointing at `orx logs`. Assisted-by: claude-opus-5.5
The log tail that wakes an agent when its run ends was mostly `::orx::` progress records: an instrumented run left 7 human lines of its last 46. The tail now leaves the records out, reading up to 64 KiB back to find the last 40 lines a person would read (still at most 4 KiB), and says how many records it left out and that `orx logs RUN` has the whole log. Assisted-by: claude-opus-5.5
Nothing named a run whose every step took three times as long as its
parent's (`slow-3x`): its numbers were level, and "2m left" sat beside a
parent that ran 1m10s in all. Once four slices of a stage are measured,
a step that takes at least twice as long as one of the run that timed
the stage (the experiment's own last run, else the parent) is said in
the strip, on the page and in `orx exp status`: "3× slower than the
parent at this stage". It compares time per step, so a stage that only
has more steps is not slower; it is derived on every evaluation from
the stage times the estimator already has, never stored. The monitor
carries it as `pace: {times, than}`, rounded to a half.
Assisted-by: claude-opus-5.5
Each stage counts its steps from zero, but a series held every stage's points on one axis, and a value reported in a stage without steps sat at its time in seconds: a parent that reported its final loss in eval drew a ghost that jumped back to x≈0, and while a child downloaded its three shards the page drew the parent's train loss across that 0–3 axis. The fold now keeps one stage's points per series on the step axis (a series starts over when a later stage reports it, recorded as `seriesStages` in the checkpoint), and once a run counts steps a value from a stage without them is only the latest. The parent's line goes under the child's only when both are the same stage, or before the child has a point, the stage it counts steps in now. A checkpoint from before reads with its stages unknown and draws as it did. Assisted-by: claude-opus-5.5
A run that hung right after its stage's first report (`hang`: train step 0, then "compiling model" and silence) was never placed in time: the fold anchors `t` to the clock only at a report that advanced, and a stage's first report advances nothing, so the estimate counted from "now" forever and no stall could be measured, even with an earlier run's pace to go by. The first report of a stage now anchors too. Its cadence is unchanged: the gap after it still never counts as one. Assisted-by: claude-opus-5.5
An open strip keeps its monitor current through events, and their patch left out `logError`, so a strip that watched a run fail never showed why: the summary had it at once, the strip's monitor never did (the `crash` fault). Squash into "fix: say why a failed run failed". Assisted-by: claude-opus-5.5
An agent reads orx-experiment-runs before a baseline's first launch and whenever it changes how a run is watched. It teaches a run's whole lifetime in one home: read the entry point, declare every stage in a plan (--config for totals known now, orx_run.plan for totals the run learns), report progress and the deciding measures, write the view and choose per experiment how runs compare (vs), what the parent shadows (ghost) and the axis (x-range), patterns for logs the agent cannot edit, the first-minutes check, and what a mid-run view, pattern or plan change reaches. The name pairs with orx-experiment-tree (the tree; its runs) and avoids `orx runs`, the command that lists them. It replaces orx-experiment-tree/references/instrument.md and the reporting section of orx-compute, which now point at it; the helper stays at compute/assets/orx_run.py, where runs already vendor it from. The text test now also compiles every documented plan through the --config path and folds every documented ::orx:: line. Assisted-by: claude-opus-5.5
A cheap agent (claude-haiku-5-5), asked only to run a baseline and two learning-rate children of a small numpy project, did each thing the skill teaches, but left what it was unsure of to the user's eye. Five attempts moved the text where it went wrong: it blocked its turn on `orx exp wait` so a mid-run request waited for the run to end; it read its own script's log with patterns rather than reporting from it, which left the eval stage without progress; it drew a row per class but never stepped the stage; it left table columns unlabelled and put two scales on one chart. Assisted-by: claude-opus-5.5
Assisted-by: claude-opus-5.5
An agent declares the rules a run is stopped or asked by beside its stage
plan, with `orx exp view set EXP --config PATH` and `{"rules": [...]}`.
Each rule is {name, text, does: stop|ask, when}, and `when` is one bounded
condition, never an expression: nonFinite, compare, noImprovement,
worseThanParent, noProgress, costAbove and projectedFinishPastLimit, with
reports where a run of reports counts. Every mistake is refused at save in
its own words, and a condition a newer orx wrote stops a launch with what to
do.
Children start with a copy, as with the view. A run freezes the rules it
was launched with in `runs.monitor_config_json`, so editing the experiment
never changes what a running run checks. `orx exp run --budget USD` adds a
rule named budget, which the page words itself since orx wrote it.
The monitor lists a run's rules and its budget, and `orx exp status` gains a
`rules` line. Nothing checks them yet.
Assisted-by: claude-opus-5.5
The first cancel accepted for a run now keeps who asked for it, in
`runs.stop_context_json`, written in the same statement that accepts the
cancel under the run's cancel lock: a user, an agent, or (next) a rule.
A later stop of any kind, and a legacy cancel that came first, never
replace it, so a racing user and rule cannot rewrite the first actor. An
old cancellation has no one recorded and reads as unknown, never guessed
from its text.
`orx exp cancel --reason TEXT` keeps the agent's words (1 to 2,048
bytes). A cancel from an agent's chat is the agent's; one with no chat,
from the dashboard or a terminal, is the user's. A cancelled `Run` gains
`outcome: {by, at, reason?, rule?}` in both run projections, and
`orx exp status` a `stop` line in the page's leads: "Stopped by the
user.", "Auto-stopped by the agent: …".
Assisted-by: claude-opus-5.5
`orx up` checks each running run's frozen rules in the monitor's pass, after it projects the run: never in the CLI, a host or the page. A pass counts the reports a run made since the last, by where they fall on the signal's axis (never polls), then trips a rule only if its condition holds once they are counted. So a NaN the run already recovered from, read in one batch after a sleep, stops nothing, and a run that went back to an earlier step counts its newest report. Nothing is checked while contact, the data or a stop already under way make the run's state unknown. What each rule counted, and that it tripped, lives in run_monitor_state's ruleState and is written in one transaction with the rule's event (a new run_monitor_events table), the notices an `ask` wakes the run's chats with, and a `stop`'s cancel and stop context under the cancel lock. A restart neither counts a report twice nor acts twice. The stop goes through Engine::accept_stop, so the existing delivery and recovery carry it to the backend, and the first stop wins. Rules on the estimate (cost past a budget, the end past the time limit) act only once four slices give a pace and two revisions agree; spend already past a budget stops at once. Until flags arrive, an `ask` wakes the chat's agent with the rule, what orx saw, and how to stop the run. Assisted-by: claude-opus-5.5
`orx exp revise EXP --run RUN` is the agent's explicit word that it is
repairing an experiment after the run stopped. From the agent's chat
during its turn it records a revision event `{runId, chatSessionId,
turnId, at}` against the experiment's newest run, if that run was
stopped; each other case (no chat, another experiment's run, a run still
going or one that ended otherwise, an older run, no turn going) is refused
in its own words.
Revising is derived, never stored: it holds while that turn is going and no
run of the experiment has started since, and ends by itself when either
changes. The summary's tone and the monitor's `revising` say it, and
`orx exp status` gains a `revising` line. The monitor task keeps such a run
watched, so its line hears at once when the revision ends; the route
projects it before answering, so lists say it straight away.
Clover has not yet said yes to the command. It is unshipped and reversible.
Assisted-by: claude-opus-5.5
The page lists the rules a run was launched with in a disclosure of its own, "3 rules, checked with no model": each rule's name, the agent's words for it, and whether it stops the run or wakes the agent, with the rule that stopped this run marked. The budget orx adds from `--budget` has no agent words, so the page words it from its amount. An Auto-stopped run's note gives the rule that stopped it in the same words, and a stop by the agent gives the agent's `--reason` when it gave one, else says the reason is in the chat. The run's types follow the server: a rule has no id, its text may be null, and an outcome carries a reason. Assisted-by: claude-opus-5.5
The orx-experiment-runs skill gains a short section on rules: when one is worth declaring, a worked config the agent-text test compiles, each condition in a table, what stop and ask do, and `--budget`. It also says to give `orx exp cancel` a `--reason` and to call `orx exp revise` before repairing an experiment whose run answered nothing; orx-compute's cancel line shows the reason. Assisted-by: claude-opus-5.5
The strip drew "Claude is revising it and will relaunch." with the running-tool shimmer, whose gradient passes through the muted ink: the first rendered check of a revising strip measured it below 4.5:1. The note now reads at full contrast; the purple mark still pulses. An end-to-end test drives the whole path through the real `orx up` and a fake agent: a NaN rule declared before launch stops the run a few reports after its third NaN, the strip reads Auto-stopped with the rule's words and no model, `orx exp status` names the rule, the agent woken by the end marks it with `orx exp revise` and the strip reads Revising with no Dismiss until its turn ends, and contrast holds in both themes at each. Assisted-by: claude-opus-5.5
A rule's stored count that does not parse was taken as no count at all, so the rule would count the run's whole history again and could stop it a second time. Such a rule is now left alone, its count kept as stored. Assisted-by: claude-opus-5.5
Each place that records a stop now names who stopped it and what it adds to `StopContext::by`, instead of spelling out every empty field. Assisted-by: claude-opus-5.5
The disclosure's title already says the rules are checked with no model; the line under it now says only when they are checked. Assisted-by: claude-opus-5.5
The fault runner's baseline now declares the rules an agent would with its view (`e2e/faults/rules.json`), which every child inherits: stop on three NaN losses in a row or a loss far behind the parent's, ask when val loss falls behind or the eval collapses. The nan-400 agent marks its stopped run with `orx exp revise` when woken; a run still going at its deadline is stopped by its agent from the chat with `--reason`. The timeline records each run's outcome and the message an ask wakes the agent with, and `--shots=DIR` saves the strip, the page's lead and its rules in both themes. Assisted-by: claude-opus-5.5
One watcher per chat looks at the runs it owns and writes each run's sentence, and a summary when it holds several. The model is one function's answer: Claude Code's Haiku one-shot, Codex's GPT Luna from its own catalog, none on other agents until a pick exists (ORX_WATCH_WITH stands in for that pick in tests and the fault bed). Looks are one-shots without tools; Codex's tools are switched off by feature flags and a turn that starts one is killed and refused. Looks are due when something changes (a bound number leaves its trend, a parent comparison flips, a count moves, a stage starts or ends, pace against the parent, a new warning family), under the chat's minGapS and a budget of ladder points; the 1/2/5/10/20/30 ladder, restarted per stage and capped by maxGapS, bounds silence while output keeps coming. The digest is bounded at 16 KiB and scanned with sharing's credential predicate before it is sent. Replies are validated strictly (tokens, one bold figure, 90 graphemes filled, no literal of a moving value), with one retry. Look intents are rows written before sending; a sent look a restart finds is interrupted, never resent. Experiments take brief and watch in --config. POST /api/runs/:id/look asks for a look under the floor; shown-runs carries each chat's latest summary; orx exp status prints the sentence. Flags are recorded in the journal only, for the flag slice. Assisted-by: claude-opus-5.5
The experiment page gains a Watch section: one row per change, looks
that kept the words folded into it ("the same on 5 looks since
10:34"), flags marked, failed and interrupted looks said as such,
what the looks cost and by which model, Look now under the chat's
least gap, and what the watcher was told. A tick's tooltip says the
look's words with the numbers of its moment and the runs it covered.
Without a plan a tick stands on its stage's bar. The summary above
the composer takes the watcher's sentence when its look covered
exactly the runs live now; attention from orx still comes first.
Settings gains Watch runs.
{signal.vs} now reads as a whole comparison ("0.12 better than the
parent"), so a sentence in any language need not finish it.
The bound-number check fits a parabola to the last 14 points and
trips when two points running land three spreads off it: the fault
bed's healthy loss curve trips none, lr-x100 trips at the turn. A new
count is counted from the log's start before its sentence is kept.
e2e/faults gains a stand-in watcher (watcher.mjs, a fixed policy over
the digest), --watcher fake|real|none, a brief on wrong-depth, the
looks in each timeline, and shots.ts for screenshots.
Assisted-by: claude-opus-5.5
Running the fault bed with the real Claude Haiku showed where the
contract was stricter than it needed to be, or unclear:
- evidence names the shown lines a cited range overlaps, instead of
refusing a range that spans lines or a gap between them;
- a run's own sentence may name the run as the summary does
({r1:loss}), stored as {loss}; a summary nobody asked for is left
unread instead of failing the look;
- a step count under ten is not a moving value a literal may not
equal: "depth 4" collided with "eval task 4";
- the digest says which log lines are the head and which are new,
that a comparison is at this step, and whether the parent's values
are its last or its current ones;
- a failed Claude one-shot says its JSON result's own words.
The fault bed signs a real look in as the shell that started it and
keeps it off the sandbox's proxy.
Assisted-by: claude-opus-5.5
A run no chat follows is "none" rather than "needs a pick", and prints no watch line in orx exp status. A log a look could not read is said to be unread in the digest. One filling routine serves the journal, the digest and status. The approved monitor fixtures carry the watch's new fields; the config refusals for brief and watch, and Codex's switched-off tools, have their tests. Assisted-by: claude-opus-5.5
Every projection of a run reads what the watcher said of it, many a second for a run that prints fast. The facts are now read again only when a look settles or a count moves, with who watches rechecked every few seconds. The page's Watch section redraws only when its watch changes. Assisted-by: claude-opus-5.5
A run whose chat has no watching model reads its stage and progress as its sentence, and this build offers no model to pick, so its page shows no Watch section until it has looks. Its late arrival below the latest run also moved where the page scrolled, which the run page's table test caught. Assisted-by: claude-opus-5.5
shots.ts prints the shot run's looks and crops a tick's tooltip wide enough to read. Assisted-by: claude-opus-5.5
A look citing a long stretch of a flood was refused for citing more than twelve lines; its first twelve say where the flood began. Assisted-by: claude-opus-5.5
The dashboard e2e tsconfig types the DOM without its iterable half, so spreading a NodeList failed `tsc` in e2e/; Array.from reads the same. Assisted-by: claude-opus-5.5
A look that flags, a rule that asks and orx's own tripwires raise a flag on the run; a flag never stops it. It keeps the log lines it cites (bytes, line numbers and the log's identity) and, in the transaction that raises it, queues one wake per following Agentic chat: the wake says the watcher saw only a digest and none of the chat, cites the lines, and says how to settle it. A Copilot flag waits for the user: the strip opens by itself with the evidence and Stop run, Keep going and Ask, the page shows the same, and the chat's sidebar row stays amber until it is settled. Tripwires watch every live run, watched or not: a value that is not a number (unless a rule the run was launched with checks that signal), no progress past the estimator's stall threshold, a run with no progress that prints nothing for ten minutes, and a kind of error line the run had not printed. Looks and tripwires keep one flag open at a time; a condition is raised once a run, and never again on an experiment where someone kept going after it. Rules that ask raise a flag in place of an attention notice. A flag's state is read back from its choices and the notice and turn that carried it, never stored. A woken turn that ends without settling it, and an agent's --ask, leave it to the user. `orx exp flag RUN --flag ID --keep|--ask|--stop` and POST /api/runs/:id/flag keep one choice per request id and refuse a settled flag or an ended run's, naming the status command to read. Keep going feeds the watcher's next looks; Ask sends the look's other open flags along, as one turn. A chat past 70% of its context is compacted in the turn slot the wake already holds, and a wake whose run ended before it was claimed is withdrawn. The danger button's text takes the red ink: the Copilot strip's contrast check measured the accent at 4.24:1. The watcher's cache of what it said is keyed by data directory too, as two stores can hold the same run id. Assisted-by: claude-opus-5.5
The experiment-runs skill taught rules but not what the watcher is told or what to do when it flags. A new section shows a brief with each label and the watch gaps as one config the text test compiles, says what a look sees (a digest, never the chat), and how to settle a flag after reading the run: keep with the reason the next looks read, stop, or leave it to the user. A rule's ask now points there, and the description names the flag that wakes the agent so the skill is read then. Assisted-by: claude-opus-5.5
The fault bed's hang-mid said its log kept printing while it was silent: a scan that caught up after a look counted as growth. Growth is now the log's length moving, and a stall says the log is silent too once it has been quiet for half the stall, since the last report and the last line land moments apart. The first line of each new kind of error now outlives a look, so an error printed while another flag was open is raised once that flag is settled; before, the look that read it meanwhile made it old news. Assisted-by: claude-opus-5.5
Two refusals and one false flag from the real Haiku on the fault bed. A look that wrote "since step 20" was refused because a count read 20 at that moment: a number written right after step, epoch or round is a place in the run, and collides only with the step the run is at now. And a healthy control's last look flagged its numbers for matching the parent's to the digit: the prompt now says matching the parent is what a healthy run does, trouble only when the brief says this run should differ, so an ended run that finished as its parent did is no flag. The bias to flag when unsure stands. On control and lr-x100 with the real Haiku after this, control's six looks raised nothing and its last said it ended level with the parent. Assisted-by: claude-opus-5.5
The watcher scanned the digest's JSON, where a tab or newline before a token becomes \t or \n and hides it, a clipped line can cut a token below its length, and the retry sent the model's reply unscanned. One credential in a run's log also failed every later look of that run, its last look included. Each log line and digest field is now scanned as written, before it is clipped or encoded. A matching log line is left out (a private key from its BEGIN line to its END line) with an omission marker the model reads, and any other field reads a marker in its place. The look goes on, and its journal row tells the user once where the credential is. Flag evidence and a tripwire's error text show the marker too, and a reply that holds a credential is never sent back in a retry. Assisted-by: claude-opus-5.5
The fold restarts a series at each stage, so a finished parent holds only its last stage's points under a name. A worseThanParent rule and the versus comparison read them at the child's step whatever stage the child was in: a child in pretrain was held against the parent's finetune loss, read as worse by 4.8, and the rule the skill teaches stopped it. Both now compare only when the child's series and the parent's are of the same stage, by its name and which time of that name it is, and the chart's parent line reads the same stage function. Where the parent has no value in that stage at that step there is no comparison. Assisted-by: claude-opus-5.5
A Stop cancelled the run, which takes seconds over a backend, before it recorded its choice. A Keep from the other side could settle the flag in between: the run stopped while its flag read "Kept going by you", and the Stop answered 409 as if nothing had happened. A Stop now records its choice and accepts the run's stop with whose words it is in one transaction under the run's cancel lock, then delivers the cancel, which the engine keeps delivering if this attempt fails. A Keep or Stop on a run whose stop was already accepted, by the agent's own `orx exp cancel` or a rule, is refused as settled, so one choice wins and every surface says its words. Assisted-by: claude-opus-5.5
A run's look after it ended counted as done whatever its result, so a model timeout, a refused reply or an orx up restart mid-call left the run with no last look for good, though the last look is what catches a run that ended wrong. The last look counts as done once one answered or is under way. One that errored or was interrupted is taken once more, its reason saying so, and a second failure stands: the run's watch reports that its last look did not finish. Assisted-by: claude-opus-5.5
A look's flag on a run with a flag still open was dropped, while its journal row was kept as "Flagged" with no outcome and no buttons, and its cursor moved past what it saw, so the new trouble reached no one. The look's row is now kept as a sentence, the journal says it was noted while the open flag was, and its words join the open flag in a "noted" event, shown beside that flag's evidence in the strip. Both land in the transaction that settles the look. Assisted-by: claude-opus-5.5
Each pass, every few seconds, read every live run's log from where the last pass stopped, and once more for every pattern ever counted on it, watched or not. For a remote run with nothing new on the laptop each read was a trip to its host, so an idle run cost a read per pass per pattern, and one slow host delayed every chat's tripwires and looks. A pass now reads only the bytes the run's fold has read past, as its checkpoint says, so an idle run costs no read and a remote one reads only what collection has seen it print. Patterns caught up with the pass count the lines it just read instead of reading them again, and each read gives up after ten seconds. Assisted-by: claude-opus-5.5
A failed read of a Codex sign-in's catalog cached nothing, so every 5 s pass started `codex app-server` again and logged the failure again, and its runs said "Reading which models Codex offers" for good. A Codex chat that had not looked yet was also never read for, since only a run's owner's sign-in was. A failure is now kept with its cause and read again after 30 s, doubling up to the catalog's ten minutes; until then the run says "Couldn't read which models Codex offers" and why. Every Codex chat that follows a live run has its catalog read. Assisted-by: claude-opus-5.5
A Keep going on one run suppressed its flag's fingerprint on every later run of the experiment, and the next looks of those runs were told not to raise it. Some fingerprints say nothing run-specific (a stall at step 0, silence at the same byte), so a later run that truly hung was never flagged, a false negative the watcher is built to avoid. Keep going now settles that run's flag only. A later run with the same cause flags again, and the kept reason reaches only that run's next looks. Orchestrator ruling, for Clover to overrule. Assisted-by: claude-opus-5.5
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Experiments should be as easy as possible to track for both the Human and the Agent. Right now, you have to watch logs yourself, or ask the agent "status pls?" in order for anything to actually happen. Instead, we're going to just tell you what's going on.
(these images are prototypes, but the system does work right now)
You can expand the card to reveal detailed graphs that live update. The agent builds these before running the experience, which is great because it can add the proper instrumentation into the experiment run script.
The agent can take advantage of this as well. We will now run Haiku/Luna, or another configured cheap model, to monitor your runs for you, which can be used to let your agent revise runs midway through. So now you don't have to bump your agents to fix problems as much. A step closer to true automated research.
Criteria
Closes #473
Closes #500 (original issue says a lot but the key is just that goal and experiment should not break. likely already the case)
Closes ORX-671
Closes ORX-678