core: cold VCS location acquisition blocked the service for 53 seconds #37794

Summary

Acquiring one cold VCS-backed V2 Location on an already-running managed service blocked for 53.4 seconds. During the stall the service stopped answering normal client work, the new TUI took over a minute to reach its event stream, and recovery logic began spawning service contenders even though the original service process remained alive.

The only current timer covers the entire Location layer, so the exact slow node is unknown. The pinned source has one high-confidence candidate, synchronous eager Fff.create() during filesystem-search layer construction, but narrower timing is required before assigning the full 53.4 seconds to FFF.

Environment

• opencode version: 0.0.0-next-15834
• OS: Darwin 25.5.0, arm64
• Terminal: WarpTerminal, TERM=xterm-256color, COLORTERM=truecolor
• Shell: /bin/zsh
• Install/channel: global Bun install, next
• Managed service: one healthy, version-matched service before the cold request; the service PID did not change during the incident
• Active plugins: three local user plugins; names and paths omitted
• Location shape: VCS-backed repository, approximately 13 GiB and 618,000 files, including approximately 8.5 GiB of dependency files

Reproduction

1. Start the managed V2 service and confirm /api/health is responsive.
2. Use other TUIs long enough that the service is handling ordinary work.
3. Open a TUI on a large VCS-backed location whose Location graph is not currently cached.
4. Measure TUI launch, Location acquisition, service health responsiveness, and first event-stream connection.

This exact 53-second result was observed once. Earlier cold acquisitions of the same repository ranged from approximately 1 to 25 seconds, so the severity is intermittent but the repository-size sensitivity is repeatable enough to have produced multiple long starts.

Expected Behavior

• Opening a Location should not synchronously scan or index the full repository before the TUI can connect.
• Search-specific indexes should initialize lazily on the first search, or off the server's request-processing thread.
• A slow Location should not prevent the shared service from answering health checks and event streams for other clients.
• Startup diagnostics should identify the slow node and operation, not only an aggregate Location duration.
• Large ignored trees such as dependency directories should not dominate first-paint latency.

Actual Behavior

Sanitized UTC timeline:

18:00:51.456  TUI process starts
18:00:52.335  update check confirms the installed version is current
18:01:52.448  config watcher fallback is logged near the end of Location acquisition
18:01:52.738  location services booted, durationMs=53359
18:01:52.791  post-boot project-copy refresh starts
18:01:54.079  TUI begins its first event-stream connection
18:01:56.081  event-stream attempt fails with Transport

The managed service had already been running for more than an hour. There is no service election or replacement before the 53.4-second timer begins, and later health checks report the same service PID. This is a cold Location graph inside the incumbent service, not a 53-second daemon launch.

At the pinned revision, buildLocationServiceMap starts one timer before compiling the full Location graph and logs only after every required layer has been acquired. The graph contains configuration, plugins, MCP, filesystem search, tools, instructions, Session services, VCS, and watchers at location-services.ts:51-99.

For VCS-backed locations, FileSystemSearch calls native Fff.create() synchronously while acquiring the layer. disableContentIndexing: true and disableMmapCache: true do not make picker creation lazy. The selection logic uses this layer for every VCS Location at search.ts:235-244.

PR #36449 already implements lazy, cached FFF creation and has focused tests, but it is currently open and conflicting with v2.

Additional Context

Facts versus hypotheses

Confirmed:

• One aggregate Location acquisition took 53,359 ms.
• It happened in the existing healthy service process.
• The repository is unusually large by file count.
• Fff.create() is synchronous and eager in the running revision.
• The post-boot project-copy refresh starts after the Location timer completes, so its separate 48-second timer is not part of the 53.4 seconds.
• Only 13 durable events totaling about 0.01 MiB were committed during the combined slow interval, so high event throughput does not explain this particular startup stall.

Not yet proven:

• Which individual Location node owns most of the 53.4 seconds.
• Whether native FFF creation alone accounts for the delay.
• Whether plugin, MCP, config, VCS, watcher, database, or scheduler contention contributes.
• Whether the main thread was continuously blocked or repeatedly starved during the full interval.

Required instrumentation

Add content-free acquisition timings around each Location node or each meaningful startup phase. Include queue/scheduler delay where an Effect is forked before its work begins. Metrics should identify node names and durations without logging paths, config contents, prompts, or tool data.

For synchronous native calls, record elapsed time before and after the call and sample event-loop delay. A process sample or trace should be captured by a deterministic large-tree benchmark rather than inferred from the aggregate timer.

Acceptance criteria

• A benchmark covers a cold VCS Location with hundreds of thousands of files and a large ignored dependency tree.
• TUI transport and basic Location readiness do not wait for a filesystem search index that has not been requested.
• The first find, glob, or grep initializes FFF once; concurrent first searches share initialization; Location disposal destroys it once.
• The managed service continues answering /api/health and existing event streams while another Location initializes.
• Logs or metrics attribute Location acquisition to individual nodes/phases and expose event-loop delay without user content.
• A slow node has a bounded timeout or graceful fallback where technically possible.
• A regression compares cold readiness before and after fix(core): initialize fff lazily #36449 or its replacement on the large-tree fixture.

Related

• 2.0: managed-service restart causes reconnect herd and resource spikes #36285 tracks the broader restart/reconnect herd and previously reduced other eager Location costs.
• fix(core): initialize fff lazily #36449 is the existing lazy-FFF implementation.
• The separate post-boot 48-second project-copy timer should not be folded into this 53.4-second measurement.

All local paths, project IDs, Session data, and plugin paths have been removed from this report.