feat: experiment progress visualization and watching

Cargo.toml

@@ -7,9 +7,9 @@ repository = "https://github.com/alphaXiv/OpenResearch"
 license = "MIT"
 
 [workspace]
-members = ["tests/support", "crates/orx-host-proto", "crates/orx-host"]
+members = ["tests/support", "crates/orx-host-proto", "crates/orx-host", "crates/orx-signals"]
 exclude = ["windows/launcher"]
-default-members = [".", "tests/support", "crates/orx-host-proto", "crates/orx-host"]
+default-members = [".", "tests/support", "crates/orx-host-proto", "crates/orx-host", "crates/orx-signals"]
 
 [target.'cfg(unix)'.dev-dependencies]
 orx-test-support = { path = "tests/support" }
@@ -55,6 +55,10 @@ semver = "1"
 fd-lock = "4"
 # Splits a pasted `ssh …` command (src/jobs/ssh/hosts.rs); already in the tree via portable-pty.
 shell-words = "1"
+# The watcher's count tokens and its sentence length (src/engine/watch); both already in the tree,
+# regex through orx-signals and unicode-segmentation through the desktop window (tao).
+regex = { version = "1", default-features = false, features = ["std", "unicode-perl", "perf"] }
+unicode-segmentation = "1"
 base64 = "0.22"
 once_cell = "1"
 # Uploaded skills (.zip of a SKILL.md folder) and shared projects (.orx). Lean:
@@ -77,6 +81,7 @@ portable-pty = "0.9"
 # async fns in the Harness trait (object-safe `dyn Harness` in the registry).
 async-trait = "0.1"
 orx-host-proto = { path = "crates/orx-host-proto" }
+orx-signals = { path = "crates/orx-signals" }
 sha2 = "0.10"
 hmac = "0.12"
 # Import the Overleaf editor session cookie from a local browser store, so the

SKILL.md

@@ -98,9 +98,11 @@ group below has a module (`orx skill <name>`) with the full flags and rules.
 | `orx compute instructions show --json` | Read the machine-wide custom recipe and its revision before configuring or launching. |
 | `orx compute [--gpu <id>] [--count <n>] [--provider <name>]` / `orx compute --cpu` | List the GPU/CPU compute catalog. |
 | `orx instance create <orgId> (--gpu <id> … \| --cpu <flavor> …)` | Spin up a standalone instance in an org; see `orx-instances`. |
-| `orx exp status/run/cancel/wait/wake <localExpId>` | Inspect, run, cancel, wait on, or register a wake-up for a local experiment node. |
+| `orx exp status/run/cancel/wait/wake <localExpId>` | Inspect (with the latest run's stage, estimate and measures), run, cancel, wait on, or register a wake-up for a local experiment node. |
+| `orx exp flag <runId> --flag <flagId> (--keep \| --ask \| --stop) [--reason "<why>"]` | Settle a flag that woke you, after reading the run yourself: keep it going, leave the choice to the user, or stop it; see `orx-experiment-runs`. |
 | `orx exp move/archive/unarchive/delete <localExpId>` | Re-parent, hide, restore, or permanently delete experiment nodes; see `orx-experiment-tree`. |
 | `orx exp desc <expId> [--set "<text>" \| --stdin]` | Read or overwrite the experiment's description. |
+| `orx exp view set <expId> [--file <view.xml>] [--pattern <patterns.json>] [--config <plan.json>]` / `view get <expId>` / `view check <runId>` | Set what an experiment's runs show and the stages they go through (copied into children), print it, or check it against a run's real log; see `orx-experiment-runs`. |
 
 To **read or edit** a node's code—including diffing what a run changed—use plain
 Git in the local session worktree. See the `orx-git` module.
@@ -134,6 +136,7 @@ The detail lives in focused modules — load one with `orx skill <name>` (the bu
 list, with one-line descriptions, is printed at the end of `orx skill` output):
 
 - **orx-experiment-tree** — the experiment-tree model, the auto-research loop, and `orx exp desc`.
+- **orx-experiment-runs** — a run the user can follow: its stage plan, progress and measures, its view and comparisons, patterns, the first-minutes check, and changes mid-run.
 - **orx-create** — initialize a local project and add local experiment nodes.
 - **orx-compute** — launch and monitor runs; after resolving the backend, read its bundled reference.
 - **orx-instances** — create persistent standalone machines for manual work.

SYSTEM_PROMPT.md

@@ -44,6 +44,13 @@ When asked to run and summarize an experiment, load `orx-compute` and
 `orx-results` before launching: execution and interpretation are separate skill
 areas. Use `orx-logs` when retrieving supporting output.
 
+Before a baseline's first launch, load `orx-experiment-runs` and make the run
+followable without asking you: its stages declared, each stage's progress and
+the measures that answer the question reported, a view that shows them, and
+its first minutes checked. Children inherit these. Load it again whenever the
+user asks to change what a run shows. Use `orx exp status <expId>` for a run's
+live stage and numbers.
+
 Use `orx` as the source of truth for the experiment tree, runs, and logs. Use
 normal repository tools for code and file inspection. Use this project id
 (`{id}`) for every `orx` command that takes one.

agent-skills/orx-compute/SKILL.md

@@ -8,14 +8,24 @@ Remote backends receive that snapshot; Tinker extracts it for a local controller
 whose SDK sends model operations remotely.
 
 ```sh
-orx exp status <expId>                 # branch, parent, run command, latest run + commit
+orx exp status <expId>                 # branch, parent, run command, latest run: stage, estimate, measures
 orx compute                            # browse GPU offers across all providers
 orx compute --gpu H100_SXM --count 1   # filter GPU offers
 orx compute --cpu                      # browse CPU-only offers
 orx exp run <expId>                    # launch on the configured default
-orx exp cancel <expId>                 # cancel the in-flight run
+orx exp cancel <expId> --reason '…'    # cancel the in-flight run, saying why
 ```
 
+## Follow a run
+
+A baseline reports its stages and measures and has a view before its first
+launch; children inherit them. How to write and change them: `orx-experiment-runs`.
+
+`orx exp status <expId>` shows a run's stage, estimate and measures; `--run
+<runId> --json` gives the stored checkpoint with its coverage and diagnostics.
+Signals are advisory: confirm trouble in the raw log before changing the
+experiment.
+
 ## Inspect configuration and custom instructions first
 
 Before configuring compute or constructing a launch command:
@@ -164,3 +174,4 @@ session waiting for results.
 - Escalate after a real OOM or hopelessly slow run instead of starting with the
   largest accelerator.
 - Raise the timeout only for genuinely long runs.
+

agent-skills/orx-compute/assets/orx_run.py

@@ -0,0 +1,117 @@
+"""Advisory run signals. Vendor with `orx skill compute/assets/orx_run.py`."""
+
+import atexit
+import contextlib
+import json
+import math
+import os
+import sys
+import threading
+import time
+
+
+_lock = threading.RLock()
+# One clock per process, from its first progress(), so `t` restarts at zero only in a new process.
+_origin = None
+
+
+def _stage(total, unit):
+    """A stage's total and unit, leaving out each one not given so the phase keeps what it has."""
+    return {key: value for key, value in (("of", total), ("unit", unit)) if value is not None}
+
+
+def plan(stages, rank=None):
+    """Declares every stage the run goes through, in order, replacing any plan declared before.
+
+    Each stage is a name, or a dict with "phase" and optionally "of" and "unit":
+    plan(["install", {"phase": "train", "of": 5000, "unit": "steps"}, "eval"]).
+    A stage that runs again is listed again where it recurs, never twice in a row.
+    """
+    listed = []
+    for stage in stages:
+        stage = {"phase": stage} if isinstance(stage, str) else dict(stage)
+        name = stage.get("phase")
+        if not isinstance(name, str) or not name:
+            raise ValueError(f"each stage needs a phase name, got {stage!r}")
+        if listed and listed[-1]["phase"] == name:
+            raise ValueError(f"{name!r} is listed twice in a row; list it again only after another stage")
+        listed.append(stage)
+    if rank is None:
+        rank = int(os.environ.get("RANK") or os.environ.get("SLURM_PROCID") or "0")
+    if rank == 0:
+        with _lock:
+            sys.stdout.write("::orx:: " + json.dumps({"plan": listed}, separators=(",", ":")) + "\n")
+            sys.stdout.flush()
+
+
+class progress:
+    def __init__(self, total=None, unit=None, rank=None, interval=1.0, start=0, phase=None):
+        if rank is None:
+            rank = int(os.environ.get("RANK") or os.environ.get("SLURM_PROCID") or "0")
+        self._enabled = rank == 0
+        self._interval = max(0.0, float(interval))
+        self._pending = None
+        self._lock = _lock
+        with self._lock:
+            global _origin
+            if _origin is None:
+                _origin = time.monotonic()
+            # Naming the phase here lets a script with no shell around it start in a named stage.
+            named = {} if phase is None else {"phase": phase}
+            self._write({**named, "step": start, **_stage(total, unit)})
+        atexit.register(self.flush)
+
+    @staticmethod
+    def _metrics(metrics):
+        result = {}
+        for key, value in metrics.items():
+            if not isinstance(value, (str, int, float, bool, type(None))) and hasattr(value, "item"):
+                value = value.item()
+            json.dumps(value, allow_nan=True)
+            result[key] = value
+        return result
+
+    def _write(self, record):
+        if self._enabled:
+            record = {**record, "t": time.monotonic() - _origin}
+            sys.stdout.write("::orx:: " + json.dumps(record, allow_nan=True, separators=(",", ":")) + "\n")
+            sys.stdout.flush()
+        self._last = time.monotonic()
+
+    def flush(self):
+        with self._lock:
+            if self._pending is not None:
+                self._write(self._pending)
+                self._pending = None
+
+    def step(self, step, **metrics):
+        with self._lock:
+            metrics = self._metrics(metrics)
+            merged = {} if self._pending is None else self._pending["m"].copy()
+            merged.update({key: value for key, value in metrics.items() if value is not None})
+            self._pending = {"step": step, "m": merged}
+            nonfinite = any(isinstance(v, float) and not math.isfinite(v) or
+                            isinstance(v, str) and v in ("NaN", "Infinity", "-Infinity")
+                            for v in metrics.values())
+            if nonfinite or time.monotonic() - self._last >= self._interval:
+                self.flush()
+
+    @contextlib.contextmanager
+    def phase(self, name, total=None, unit=None):
+        with self._lock:
+            self.flush()
+            self._write({"phase": name, "step": 0, **_stage(total, unit)})
+        try:
+            yield self
+        finally:
+            self.flush()
+
+    def row(self, name, **cells):
+        with self._lock:
+            self.flush()
+            self._write({"row": name, "m": self._metrics(cells)})
+
+    def spend(self, cumulative_usd):
+        with self._lock:
+            self.flush()
+            self._write({"spend": cumulative_usd})

agent-skills/orx-create/SKILL.md

@@ -47,7 +47,11 @@ orx create-experiment <localProjectId> --title "Alternative baseline" --baseline
   the oldest root unless `--baseline` explicitly requests another root.
 - A child branches from its parent's Git branch and inherits its run
   command. Vary code or configuration on the child; keep the command fixed.
+- `--title` is what the user reads beside the node's progress among its
+  siblings: short, the setting that varies first ("Depth 4 at 2e15 FLOPs").
 - `--description` should state the concrete change and measurement the node
   owes.
+- Before a baseline's first run, declare its stages, report its measures and
+  set its view (`orx-experiment-runs`).
 - Commit the node's branch before launching. Runs use an immutable archive of
   that local commit.