Run the collector on the host, not in a container

.github/workflows/ci.yml

@@ -65,35 +65,42 @@ jobs:
         run: python3 scripts/check-dashboard.py
 
   compose:
-    name: compose files parse
+    name: compose and scripts
     runs-on: ubuntu-latest
-    # The overlays only patch services, so they are not valid on their own -
-    # they are checked the way they are actually used. GITHUB_TOKEN is a
-    # required variable; any value satisfies interpolation.
+    # GITHUB_TOKEN is a required variable in the compose file; any value
+    # satisfies interpolation.
     env:
       GITHUB_TOKEN: dummy-value-for-interpolation-only
     steps:
       - uses: actions/checkout@v4
 
+      - name: Compose files parse
+        run: |
+          docker compose -f docker-compose.yml config --quiet
+          docker compose -f docker-compose.yml -f docker-compose.admin.yml config --quiet
+
+      # The collector runs on the host now, so these are the entry points a
+      # user actually touches. A syntax error in one of them used to be caught
+      # by nothing at all.
+      - name: Shell scripts parse
+        run: for f in scripts/*.sh; do bash -n "$f"; done
+
       # repos.yml is gitignored, so CI builds one the way a new user would and
-      # proves gen-repos.py emits an override that actually merges. Nothing
-      # else in CI reads it, so a malformed one would otherwise only surface
-      # on somebody's laptop.
-      - name: Generate the fleet override
+      # proves gen-repos.py emits the two lines the collector reads - including
+      # for a checkout that does NOT sit at <root>/<owner>/<name>, which is the
+      # case the paths line exists for. Nothing else in CI reads repos.yml, so
+      # a malformed one would otherwise only surface on somebody's laptop.
+      - name: The fleet resolves to an environment
         run: |
           pip install --quiet pyyaml
           printf 'repos:\n' > repos.yml
-          for full in Jebel-Quant/rhiza cvxgrp/cvxsimulator; do
-            git init -q -b main "ci-checkouts/$full"
-            git -C "ci-checkouts/$full" remote add origin "https://github.com/$full.git"
-            printf '  - path: ci-checkouts/%s\n' "$full" >> repos.yml
-          done
-          # A repo with no checkout: GitHub panels only, no bind mount.
+          git init -q -b main nested/somewhere/rhiza
+          git -C nested/somewhere/rhiza remote add origin https://github.com/Jebel-Quant/rhiza.git
+          printf '  - path: nested/somewhere/rhiza\n' >> repos.yml
+          # A repo with no checkout: GitHub panels only, no path.
           printf '  - repo: Jebel-Quant/actions\n' >> repos.yml
-          python3 scripts/gen-repos.py
-          cat docker-compose.repos.yml
 
-      - name: Base + fleet override
-        run: docker compose -f docker-compose.yml -f docker-compose.repos.yml config --quiet
-      - name: Base + admin overlay
-        run: docker compose -f docker-compose.yml -f docker-compose.repos.yml -f docker-compose.admin.yml config --quiet
+          python3 scripts/gen-repos.py | tee env.out
+          grep -qx 'JQ_REPOS=Jebel-Quant/rhiza,Jebel-Quant/actions' env.out
+          grep -q 'JQ_REPO_PATHS=Jebel-Quant/rhiza=.*/nested/somewhere/rhiza$' env.out
+          test "$(grep -c 'Jebel-Quant/actions=' env.out)" = 0

.gitignore

@@ -1,12 +1,14 @@
 .env
 
-# Your fleet and the mounts generated from it: both describe the folder layout
-# of one machine, which has no business in a public repo. Start from
-# repos.example.yml.
+# Your fleet: it describes the folder layout of one machine, which has no
+# business in a public repo. Start from repos.example.yml. Nothing is generated
+# from it any more - scripts/collector.sh reads it at every launch.
 repos.yml
-docker-compose.repos.yml
 
-# Left behind by running the collector natively instead of in its container.
+# Where the launchd agent's output goes.
+.collector-logs/
+
+# Left behind by running the collector, which is how it runs now.
 .venv/
 .ruff_cache/
 .pytest_cache/

README.md

@@ -15,8 +15,10 @@ the board because you listed it in `repos.yml`, and for no other reason.
 
 ## What you need
 
-Docker, and the [`gh` CLI](https://cli.github.com) signed in (`up.sh` mints the
-token from it — otherwise put a `GITHUB_TOKEN` in `.env` yourself).
+Docker (for Prometheus and Grafana), [`uv`](https://docs.astral.sh/uv/) (for the
+collector, which runs on your machine rather than in a container), and the
+[`gh` CLI](https://cli.github.com) signed in — `up.sh` mints the token from it,
+otherwise put a `GITHUB_TOKEN` in `.env` yourself.
 
 ## Recipe
 
@@ -34,7 +36,9 @@ repos:
   - repo: Jebel-Quant/actions           # monitored, but not cloned here
 ```
 
-`owner/name` comes from each checkout's `origin`, so the path is all you write.
+`owner/name` comes from each checkout's `origin`, so the path is all you write,
+and the path is used as written — a checkout does not have to live at
+`<root>/<owner>/<name>`.
 Then:
 
 ```bash
@@ -43,8 +47,9 @@ open http://localhost:3000/d/jq-fleet
 ```
 
 The board fills in within a minute — the local panels first, the GitHub panels
-after the first API refresh. Both `repos.yml` and the generated
-`docker-compose.repos.yml` are gitignored: they describe one machine's folders.
+after the first API refresh. `repos.yml` is gitignored: it describes one machine's folders. Nothing is
+generated from it — the collector reads it at every launch, so there is no
+second file to fall out of step.
 
 ## Then what
 
@@ -53,6 +58,7 @@ after the first API refresh. Both `repos.yml` and the generated
 | Add or drop a repo | edit `repos.yml`, `./scripts/up.sh` again — [details](docs/configuration.md) |
 | Erase a dropped repo's history | [`./scripts/purge-repo.sh owner/name`](docs/configuration.md#dropping-a-repo) (irreversible) |
 | Stop | `./scripts/down.sh` (add `--volumes` to discard the history too) |
+| See the collector's log | `tail -f .collector-logs/collector.log` — it runs on your machine, not in Docker ([why](docs/operations.md#the-collector-runs-on-your-machine)) |
 | Edit the board | change `grafana/dashboards/fleet.json`; it reloads in 30s — [read the traps first](docs/dashboard.md#traps-worth-not-re-introducing) |
 | Get notified | add a contact point under *Alerting → Contact points* — [why it is not provisioned](docs/dashboard.md#alerting) |
 

collector/tests/test_fleet.py

@@ -436,44 +436,40 @@ def test_a_directory_that_is_not_a_checkout_fails(gen_repos, tmp_path):
         gen_repos.resolve({"path": str(tmp_path / "empty")}, 1)
 
 
-def test_env_mode_prints_the_fleet_and_the_paths_and_writes_nothing(
+def test_it_prints_the_fleet_and_the_paths_and_writes_nothing(
     gen_repos, tmp_path, monkeypatch, capsys
 ):
     """Both lines must come from repos.yml, not from anyone's memory.
 
     Retyping either is how they drift: a repo added here never reaches the
-    board and nothing reports the difference. The paths line is what lets a
-    collector running outside the container find a checkout that does not sit
-    at <root>/<owner>/<name> - and a repo with no checkout must not appear in
-    it at all. --env must also leave the compose override alone, so it is safe
-    to run on a machine that has no stack.
+    board and nothing reports the difference. The paths line is what lets the
+    collector find a checkout that does not sit at <root>/<owner>/<name>, and a
+    repo with no checkout must not appear in it at all. Nothing is written to
+    disk - scripts/collector.sh runs this at every launch and exports the
+    result, so there is no generated file in between to go stale.
     """
     path = make_checkout(tmp_path, "Jebel-Quant", "rhiza")
     source = tmp_path / "repos.yml"
     source.write_text(f"repos:\n  - path: {path}\n  - repo: cvxgrp/cvxsimulator\n")
-    target = tmp_path / "docker-compose.repos.yml"
     monkeypatch.setattr(gen_repos, "SOURCE", source)
-    monkeypatch.setattr(gen_repos, "TARGET", target)
-    monkeypatch.setattr(sys, "argv", ["gen-repos.py", "--env"])
+    monkeypatch.setattr(sys, "argv", ["gen-repos.py"])
 
     gen_repos.main()
 
-    lines = capsys.readouterr().out.strip().splitlines()
-    assert lines == [
+    assert capsys.readouterr().out.strip().splitlines() == [
         "JQ_REPOS=Jebel-Quant/rhiza,cvxgrp/cvxsimulator",
         f"JQ_REPO_PATHS=Jebel-Quant/rhiza={path}",
     ]
-    assert not target.exists()
+    assert list(tmp_path.glob("docker-compose*")) == []
 
 
-def test_env_mode_refuses_a_path_it_cannot_express(gen_repos, tmp_path, monkeypatch, capsys):
+def test_it_refuses_a_path_it_cannot_express(gen_repos, tmp_path, monkeypatch, capsys):
     """A comma is the separator, so a path holding one would read as two repos."""
     path = make_checkout(tmp_path, "Jebel-Quant", "rhi,za")
     source = tmp_path / "repos.yml"
     source.write_text(f"repos:\n  - path: {path}\n")
     monkeypatch.setattr(gen_repos, "SOURCE", source)
-    monkeypatch.setattr(gen_repos, "TARGET", tmp_path / "out.yml")
-    monkeypatch.setattr(sys, "argv", ["gen-repos.py", "--env"])
+    monkeypatch.setattr(sys, "argv", ["gen-repos.py"])
 
     with pytest.raises(SystemExit):
         gen_repos.main()

docker-compose.yml

@@ -1,37 +1,15 @@
-# The laptop stack. Not usable on its own - the fleet lives in repos.yml,
-# so bring it up with scripts/up.sh, or with:
+# Prometheus and Grafana. The collector is NOT here: it runs on the host, via
+# scripts/collector.sh, because it reads your working copies and a container can
+# only see them through bind mounts - which cannot name a checkout that sits
+# somewhere other than <root>/<owner>/<name>, and which are slow enough on
+# macOS to have needed a caching layer to work around.
 #
-#   python3 scripts/gen-repos.py
-#   docker compose -f docker-compose.yml -f docker-compose.repos.yml up -d
+#   ./scripts/up.sh          starts both halves
+#   docker compose up -d     starts only this half
 
 name: jq-monitoring
 
 services:
-  collector:
-    build: ./collector
-    container_name: jq-collector
-    restart: unless-stopped
-    environment:
-      GITHUB_TOKEN: ${GITHUB_TOKEN:?set GITHUB_TOKEN in monitoring/.env - scripts/up.sh does it for you}
-      # JQ_REPOS and the per-repo bind mounts both come from
-      # docker-compose.repos.yml, generated from repos.yml. This file has no
-      # opinion about which repos are yours.
-      JQ_REPO_ROOT: /repos
-      JQ_TEMPLATE_REPO: ${JQ_TEMPLATE_REPO:-Jebel-Quant/rhiza}
-      JQ_GITHUB_INTERVAL: ${JQ_GITHUB_INTERVAL:-600}
-      JQ_LOCAL_INTERVAL: ${JQ_LOCAL_INTERVAL:-60}
-      JQ_MEASURE_MAX_AGE: ${JQ_MEASURE_MAX_AGE:-86400}
-      JQ_IGNORE: ${JQ_IGNORE:-}
-      JQ_PUBLIC_ONLY: ${JQ_PUBLIC_ONLY:-false}
-      JQ_LOG_LEVEL: ${JQ_LOG_LEVEL:-INFO}
-    # No volumes here on purpose. Every checkout is mounted individually by
-    # docker-compose.repos.yml, read-only: the collector reports on your working
-    # copies and must never be able to touch them, and mounting only what is
-    # listed means an unlisted repo is not merely filtered out - it is not even
-    # visible to the container. Every git call is --no-optional-locks too.
-    ports:
-      - "127.0.0.1:9109:9109"
-
   prometheus:
     image: prom/prometheus:v2.55.1
     container_name: jq-prometheus
@@ -46,8 +24,10 @@ services:
       - prometheus-data:/prometheus
     ports:
       - "127.0.0.1:9090:9090"
-    depends_on:
-      - collector
+    # Docker Desktop resolves host.docker.internal on its own; plain Docker on
+    # Linux does not, and without this the collector target is simply down.
+    extra_hosts:
+      - "host.docker.internal:host-gateway"
 
   grafana:
     image: grafana/grafana:11.3.1

docs/configuration.md

@@ -87,27 +87,23 @@ better a refusal than a board that is quietly one repo short.
 | `JQ_IGNORE` | — | Repos to drop without editing `repos.yml`, as bare names or `owner/name`. Applies to both halves. |
 | `JQ_INCLUDE_ARCHIVED` | `false` | Archived repos are dropped from both halves. |
 | `JQ_PUBLIC_ONLY` | `false` | Drop private repos entirely — not just their details, their existence. |
-| `JQ_REPO_PATHS` | — | Where each checkout really is, as `owner/name=path` pairs. Only needed outside the container — see [Checkout paths](#checkout-paths). |
+| `JQ_REPO_PATHS` | — | Where each checkout really is, as `owner/name=path` pairs. `scripts/collector.sh` sets it from `repos.yml` — see [Checkout paths](#checkout-paths). |
 | `JQ_GITHUB_INTERVAL` | `300` | Seconds between GitHub refreshes. |
 | `JQ_MEASURE_MAX_AGE` | `86400` | Seconds an unchanged line/commit count may stand before it is retaken. See [Size and cadence](dashboard.md#size-and-cadence). |
 | `PROM_RETENTION` | `180d` | How much history to keep. |
 
-`scripts/gen-repos.py` turns `repos.yml` into the `JQ_REPOS` list the
-collector actually reads, so both halves see the same fleet. Setting `JQ_REPOS`
-by hand works too, and skips `repos.yml` entirely — but then nothing mounts the
-checkouts, so only the GitHub panels have anything to say.
+`scripts/gen-repos.py` turns `repos.yml` into the `JQ_REPOS` list the collector
+actually reads, so both halves see the same fleet. Setting `JQ_REPOS` by hand
+works too and skips `repos.yml` entirely — but then no checkout paths are set
+either, so only the GitHub panels have anything to say.
 
 ## Checkout paths
 
 The collector reads a repo at the path `JQ_REPO_PATHS` gives for it, and failing
 that at `<JQ_REPO_ROOT>/<owner>/<name>`.
 
-**In the container only the fallback is used, and it is always right**, because
-the generated bind mounts put every checkout at exactly that path. `repos.yml`
-can say anything and the mount normalises it.
-
-**Outside the container it is often wrong**, because paths are whatever they are
-on disk. An entry like
+The collector runs on your machine, so paths are whatever they are on disk and
+the fallback is often wrong. An entry like
 
 ```yaml
   - path: ~/repos/tschm/rhiza_projects/cs     # this is tschm/cs
@@ -118,11 +114,13 @@ working-copy panels while staying on the GitHub ones — present on the board, a
 quietly missing half its columns. Four repos in the fleet this was built against
 are laid out that way.
 
-So when you run the collector on the host, take both lines from `repos.yml`
-rather than writing them out:
+You do not normally set this yourself: `scripts/collector.sh` runs
+`scripts/gen-repos.py` at every launch and exports both lines, so `repos.yml`
+stays the only place the fleet and the layout are written down. To see what it
+resolves to:
 
 ```bash
-uv run --with pyyaml python scripts/gen-repos.py --env >> .env
+uv run --with pyyaml python scripts/gen-repos.py
 ```
 
 A path containing a comma cannot be expressed — comma is the separator — and

docs/dashboard.md

@@ -107,6 +107,14 @@ name to `owner/name` when the cvxgrp repos were added, so history from before
 that point lives under the old names. Windows spanning the change show both
 generations; they age out. This is inherent to Prometheus, not a misconfiguration.
 
+**The `instance` label is pinned on purpose.** There is one collector, so
+`instance` says only where the process happened to be running — and left alone
+it is the scrape address. When the collector moved out of its container the
+address changed, every series forked in two, and for one stale window each repo
+appeared on the board twice. `prometheus.yml` relabels it to a constant so that
+cannot recur; series from before the move keep the old value and age out with
+retention.
+
 **Never `group_left` onto an `_info` metric without a `topk` guard.** The
 `_info` metrics carry a label that changes value — `ref`, `branch`,
 `conclusion`, `checks`. When one changes, Prometheus returns the old *and* the

docs/operations.md

@@ -6,6 +6,42 @@ keywords: grafana login, no data, prometheus staleness, macos sleep, caffeinate
 
 # Running it day to day
 
+## The collector runs on your machine
+
+Two halves, not three containers. Prometheus and Grafana are in Docker;
+**the collector is an ordinary process on your Mac**, started by `up.sh` as a
+launchd agent and stopped by `down.sh`.
+
+It is not containerised because it reads your working copies, and a container
+can only reach those through bind mounts. Those mounts have to place every
+checkout at `<root>/<owner>/<name>`, which silently loses any repo that lives
+somewhere else, and reading thousands of small files back through them on macOS
+is slow enough that the line counts needed a cache to stay affordable. On the
+host both problems disappear.
+
+| | |
+|---|---|
+| Its log | `.collector-logs/collector.log` |
+| Restart it | `launchctl kickstart -k gui/$UID/com.jebel-quant.jq-collector` |
+| Run it in the foreground instead | `./scripts/collector.sh` (Ctrl-C stops it) |
+| Is Prometheus reaching it? | <http://localhost:9090/targets> — the `jq-collector` job |
+
+Prometheus scrapes it at `host.docker.internal:9109`, which is how a container
+reaches the machine it runs on.
+
+**The trade.** Inside the container the collector could only see the checkouts
+mounted into it, so an unlisted repo was not merely filtered out — it was
+invisible. It now runs as you and could read anything you can. It still never
+writes: every git call is read-only and passes `--no-optional-locks`. But that
+is now a property of the code rather than something the sandbox enforces.
+
+**If it will not start**, the usual cause is `PATH`. A launchd agent inherits
+`/usr/bin:/bin:/usr/sbin:/sbin` and nothing else, so a `uv` under
+`/opt/homebrew` or `~/.local` is invisible to it. `up.sh` pins `uv`'s directory
+into the plist when it installs the agent, so re-running `./scripts/up.sh`
+after moving or reinstalling `uv` is the fix.
+
+
 ## The "Sign in" button
 
 Grafana's own local login, against a SQLite file in the `grafana-data` volume on

prometheus/prometheus.yml

@@ -9,8 +9,22 @@ global:
 
 scrape_configs:
   - job_name: jq-collector
+    # The collector runs on the host, not in this stack - it reads your working
+    # copies at their real paths, which a bind mount cannot express and which
+    # is far faster than one anyway. host.docker.internal is how a container
+    # reaches the machine it runs on; docker-compose.yml maps it on Linux,
+    # where it is not built in.
     static_configs:
-      - targets: ["collector:9109"]
+      - targets: ["host.docker.internal:9109"]
+    relabel_configs:
+      # There is exactly one collector, so `instance` carries no information -
+      # only where the process happened to be running. Left alone it is the
+      # scrape address, and moving the collector (container -> host, or back)
+      # silently forks every series in two: the board then joins a repo against
+      # itself and shows each one twice until the old series go stale. Pin it,
+      # and the history stays continuous across the move.
+      - target_label: instance
+        replacement: jq-collector
 
   - job_name: prometheus
     static_configs: