[Search Indexing] Add distributed load-testing harness with ramp testing, health monitoring, bottleneck analysis #27585

Problem

There was no repeatable load-testing harness for reindex or general server behavior. Performance work relied on ad-hoc scripts, and it was difficult to measure the impact of changes like client upgrades, batching tweaks, or executor changes with statistical confidence. Finding the saturation point of a cluster — or proving a change moved it — required hand-rolled tooling each time.

Impact

• Performance regressions caught late, often in production.
• No shared understanding of what a "healthy" reindex looks like at different entity scales.
• Difficult to correlate diagnostic snapshots with the workload that produced them.

Fix

Added a full distributed load-testing harness.

• Location: bin/distributed-test/
• scripts/perf-test.sh — data generation across 30+ entity types with phase-based creation (metadata → services → infrastructure → core entities → data quality → lineage → time-series). Scale presets: 10k through 500k, plus small, medium, large, xlarge.
• HealthMonitor — polls /api/v1/system/version every 5s and /api/v1/system/diagnostics every 10s during the run.
• ServerIntrospector — pre/post-load snapshots (version, component health, entity counts, service counts, Prometheus scrape, diagnostics).
• RampTester — doubling concurrency (1, 2, 4, 8, …) to find optimal worker count and saturation. Flags saturation when p95 exceeds 3× the prior level OR errors appear.
• JSON report per run (benchmark-report-{YYYYMMDD-HHMMSS}.json):
  • Entity benchmarks with latency percentiles (p50 / p75 / p90 / p95 / p99), throughput RPS, error rate, error samples.
  • Latency analysis: bimodal detection (p90 / p50 > 5), tail-latency (p99 / p95 > 3), degradation tracking.
  • Throughput windows (5-second buckets).
  • Cluster sizing recommendations.
• Docs: bin/distributed-test/DIAGNOSTICS.md, bin/distributed-test/USAGE.md.

Parent

Epic: #27577

Resolved by

PR #26154