Review fixes, Chromium worktree benchmark, and managed attach performance (#210)

.github/workflows/benchmark-chromium-worktrees.yml

@@ -0,0 +1,80 @@
+name: Benchmark Chromium worktrees
+
+on:
+  workflow_dispatch:
+    inputs:
+      worktrees:
+        description: Linked worktrees to add after the primary checkout (at least 2)
+        default: "3"
+      repeat:
+        description: Timed rounds per query and target
+        default: "3"
+      dirty_files:
+        description: Tracked files to modify in the last worktree before attaching it
+        default: "16"
+
+permissions:
+  contents: read
+
+env:
+  CARGO_TERM_COLOR: always
+
+jobs:
+  benchmark:
+    name: Chromium worktrees (${{ matrix.os }})
+    timeout-minutes: 360
+    strategy:
+      fail-fast: false
+      matrix:
+        include:
+          - os: ubuntu-latest
+            python: python3
+          - os: macos-latest
+            python: python3
+          - os: windows-latest
+            python: python
+    runs-on: ${{ matrix.os }}
+    steps:
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+
+      - name: Build release
+        run: cargo build --release
+
+      - name: Free disk space (Linux)
+        if: runner.os == 'Linux'
+        run: |
+          sudo rm -rf /usr/share/dotnet /usr/local/lib/android /opt/ghc /opt/hostedtoolcache
+          df -h
+
+      # The first worktree shares the primary commit (generation reuse), the
+      # second is one commit older (a new generation), and the last is dirty.
+      - name: Run benchmark
+        shell: bash
+        env:
+          WORKTREES: ${{ inputs.worktrees }}
+          REPEAT: ${{ inputs.repeat }}
+          DIRTY_FILES: ${{ inputs.dirty_files }}
+        run: |
+          ${{ matrix.python }} -B scripts/benchmark_worktrees.py \
+            --bench-dir "$RUNNER_TEMP/tgrep-worktree-bench" \
+            --worktrees "$WORKTREES" \
+            --worktree-rev HEAD --worktree-rev HEAD~1 \
+            --dirty-files "$DIRTY_FILES" \
+            --repeat "$REPEAT" \
+            --results-json worktree-benchmark.json \
+            --results-md worktree-benchmark.md
+
+      - name: Print results
+        if: always()
+        shell: bash
+        run: cat worktree-benchmark.md || true
+
+      - name: Upload results
+        if: always()
+        uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
+        with:
+          name: benchmark-chromium-worktrees-${{ matrix.os }}
+          path: |
+            worktree-benchmark.json
+            worktree-benchmark.md
+          if-no-files-found: ignore

AGENTS.md

@@ -101,17 +101,23 @@ Legacy/unmanaged v1 storage remains retain-all.
 Managed attachments use bounded parallel verified-read batches without raising
 namespace operation-worker limits. Inspect `last_reconcile.timings` and its
 read/reuse/resource counters to distinguish queueing, discovery, verification,
-catalog/control and checkpoint work. Healthy native managed views also get a
+catalog/control and checkpoint work. Managed views are eventually consistent:
+after their first publication they report `serving: true` and keep answering
+queries from the last published snapshot while later edits are reconciled, so
+results may lag recent edits. `ready: true` additionally means the view has
+caught up with every observed invalidation. Healthy native managed views also get a
 periodic full verification every `--poll-interval` seconds (default 120) while
 they keep serving. It rereads all admitted contents under the normal operation
-budgets, and closes readiness only when it finds a difference, then publishes
+budgets, and closes `ready` only when it finds a difference, then publishes
 the repair. A deadline or cancellation leaves the view ready but unverified
 (`last_verification.outcome` `failed`/`cancelled`) until the next interval, so
 results may lag missed changes by one interval plus verification time.
 `views.refresh` explicitly verifies all admitted contents; use `--no-index`
 when current disk bytes are required.
 `--watch-mode poll` and actual native-watcher failure/budget fallback instead
-close readiness for full verification after `--poll-interval` seconds.
+close `ready` for full verification after `--poll-interval` seconds.
+`watch_events` in status counts watcher input by kind, metadata-only
+notifications filtered as unchanged, invalidated paths and epoch advances.
 `reconciliation_policy` distinguishes configured mode from each view's effective
 polling and reports namespace monitor counts. First attachment/restore and
 reported watcher errors/rescans retain full verification. A clean Git status
@@ -230,10 +236,10 @@ tgrep --json -F -- "fn main" tgrep-cli/build.rs
 {"data":{"elapsed_total":{"human":"0.002651s","nanos":2651400,"secs":0},"stats":{"bytes_printed":260,"bytes_searched":1775,"elapsed":{"human":"0.002651s","nanos":2651400,"secs":0},"matched_lines":1,"matches":1,"searches":1,"searches_with_match":1}},"type":"summary"}
 ```
 
-Any parser written for `rg --json` works as is, with one exception: on a line
-that is not valid UTF-8, ripgrep emits base64 `lines.bytes`, while tgrep
-always emits `lines.text` with each bad byte replaced by U+FFFD. A consumer
-that depends on the raw bytes of such lines will see repaired text instead.
+Any parser written for `rg --json` works as is. As in ripgrep, `lines.text`
+keeps each line's own terminator (`\r\n` for CRLF files), and a line that is not
+valid UTF-8 is emitted as base64 `lines.bytes` with submatch offsets into those
+raw bytes. Paths are always `path.text`.
 
 `--vimgrep` gives `file:line:col:text`, one row per match, which is the
 easiest format to feed into a "jump to location" step.

CONTRIBUTING.md

@@ -36,10 +36,10 @@ cargo build --release
 
 After the version bump is merged and CI passes, open **Actions > Release >
 Run workflow**, select **main**, and enter the tag matching the workspace
-version in `Cargo.toml`, for example `v1.2.3`. The equivalent CLI command is:
+version in `Cargo.toml`, for example `v1.2.4`. The equivalent CLI command is:
 
 ```bash
-gh workflow run release.yml --repo microsoft/tgrep --ref main -f tag=v1.2.3
+gh workflow run release.yml --repo microsoft/tgrep --ref main -f tag=v1.2.4
 ```
 
 The workflow validates the version, creates a missing tag at the exact commit

Cargo.toml

@@ -4,7 +4,7 @@ exclude = ["vendor/ignore"]
 resolver = "2"
 
 [workspace.package]
-version = "1.2.3"
+version = "1.2.4"
 edition = "2024"
 rust-version = "1.88"
 license = "MIT"

README.md

@@ -225,6 +225,10 @@ including nested repositories' Git internals. `--hidden` does not override this;
 use `--no-ignore` consistently on `index` and `serve` to include them. Explicit
 filesystem scans (`--no-index`) retain ripgrep-style hidden and ignore behavior.
 Git metadata needed for ignore rules and tracked-file detection is still read.
+As in ripgrep, a file a scan cannot read is reported and makes the exit code 2.
+On Windows a shared daemon holds a byte-range lock on
+`.git/tgrep-daemon-v1.lock`, so `--no-index --hidden` scans of that checkout
+report it; add `-g '!.git'` when Git internals are not wanted.
 
 On Windows, replaced index generations stay in `.retired` while readers have
 them memory-mapped. Cleanup retries after publication, every minute, and on
@@ -426,8 +430,8 @@ These tables describe search flags. Use `tgrep index --help` and
 | `--json` | ripgrep-compatible JSON stream (one object per line) |
 | `--vimgrep` | Vim-compatible `file:line:col:content`, one row per match |
 | `--color auto/always/never` | Color mode control |
-| `-0, --null` | NUL byte filename separator (for xargs) |
-| `--trim` | Trim leading/trailing whitespace |
+| `-0, --null` | Follow every printed file path with a NUL byte instead of its separator, in all output modes except JSON (for xargs) |
+| `--trim` | Trim leading ASCII whitespace from printed lines (JSON lines are unchanged) |
 | `-., --hidden` | Include hidden files and directories |
 | `--no-ignore` | Don't respect `.gitignore` or `p4ignore.ini` files |
 | `-a, --text` | Search binary files as if they were text |
@@ -662,6 +666,8 @@ matching file, rather than transmitting every matching line, span and column.
 `--stats` still reports the original match and matched-line totals; it does not
 restore unused line detail. Counts therefore do not consume reply space
 proportional to matching content, but remain subject to the same response limit.
+Server `-l` returns one path row per file and, without `--stats`, stops at the
+first match in each file.
 Ordinary older servers that ignore `-c` projection retain line-row counting.
 Shared servers that reject the new option, and servers without metadata needed
 for `--count-matches`, fall back with a diagnostic. Malformed projected counts
@@ -762,8 +768,11 @@ UTF-8 sequences with `U+FFFD`. A pattern can match those replacement characters.
 For UTF-8 repair, text-output columns and byte offsets are mapped back to the
 source bytes.
 
-JSON output uses `lines.text` with replacements and submatch offsets into that
-text. ripgrep instead emits base64 `lines.bytes` for invalid UTF-8.
+JSON output follows ripgrep for invalid UTF-8: such a line is emitted as base64
+`lines.bytes` of the original source bytes, with submatch offsets into those
+bytes and each submatch as `text` when valid UTF-8, otherwise `bytes`.
+`lines.text` keeps each line's own terminator, including `\r\n`.
+Paths are always emitted as `path.text`.
 Transcoded encodings use decoded-text offsets rather than original byte
 positions; offsets also exclude stripped BOM bytes.
 
@@ -929,7 +938,13 @@ index/reconcile worker, two independent query workers, bounded connection and
 work queues, 1 MiB requests and 64 MiB responses. There is no content cache.
 The response limit is enforced while producing individual match/context rows,
 file paths and statistics, including JSON escaping and envelope overhead, and
-again during final serialization. Large queries fail explicitly and CLI queries scan;
+again during final serialization. Daemons advertising `paged_results` (v1) or
+the `paged-results` capability (managed v2) split a large search or `--files`
+result into pages of at most that size, ending a page after about a minute of
+work; clients request each `next_page` cursor in path order and merge the rows.
+Each page reads the then-current snapshot, so a multi-page result is
+eventually consistent. Only a single file whose rows exceed the limit, or a
+daemon without paging, fails explicitly, and CLI queries then scan;
 generation mappings and private posting memory still scale with admitted data.
 
 Storage must already exist, be trusted and outside **all** registered worktrees,
@@ -1020,15 +1035,21 @@ Healthy native managed views also run a periodic full verification every
 `--poll-interval` seconds (default 120), measured from their last completed
 full pass, while they keep serving queries. It reads and hashes every admitted
 file under the existing operation and resource budgets. An unchanged result
-keeps the view ready and changes nothing. Readiness closes only when it finds a
+keeps the view ready and changes nothing. `ready` closes only when it finds a
 difference, after which it publishes a full repair; silently missed changes
 are therefore repaired within one interval plus the verification time.
 Watcher errors/rescans, explicit full invalidation and concurrent edits still
-close readiness first. A verification that exhausts its deadline or is
+close `ready` first. Managed views are eventually consistent: once published
+they report `serving: true` and keep answering from their last publication
+while `ready` is closed, instead of sending clients to a filesystem scan.
+Results can lag recent edits until the next reconciliation publishes. A
+verification that exhausts its deadline or is
 cancelled leaves the view ready but unverified, reports `failed`/`cancelled`,
 and retries after the next interval. Timestamps, clean Git status and an
 unavailable journal never prove that nothing changed. Status reports
-`last_verification` with its outcome and age. Use `--no-index` for current disk bytes.
+`last_verification` with its outcome and age, and `watch_events` counts
+watcher input by kind, metadata-only notifications filtered as unchanged,
+invalidated paths and epoch advances. Use `--no-index` for current disk bytes.
 
 On Windows/macOS, managed views use recursive worktree and Git metadata-root
 subscriptions, without pre-enumerating descendant directories. The unchanged
@@ -1044,8 +1065,9 @@ Explicit `--watch-mode poll` and actual native-watcher failure/budget fallback
 still perform full reconciliation after `--poll-interval` seconds (default 120)
 following completion. `--no-watch` disables watching and polling, not initial
 verification or explicit refresh. `reconciliation_policy` distinguishes these
-per-view modes and namespace monitor counts. Polling views close readiness for
-each pass; healthy native views keep serving through theirs. These
+per-view modes and namespace monitor counts. Polling views close `ready` for
+each pass but keep serving their previous publication; healthy native views
+stay ready through theirs. These
 recurring-mode changes postdate the immutable benchmark source and are not
 measured there.
 Legacy v1 and ordinary-server refresh policies are unchanged.