Skip to content

perf(rpc): goals bulkAnalytics fires 2 ClickHouse queries per goal (N+1) #679

Description

@FindMalek

Problem

goalsRouter.bulkAnalytics (packages/rpc/src/routers/goals.ts) fires two ClickHouse queries per goal — one for the conversion count (processGoalConversionCount) and one for the total-website-users denominator (getTotalWebsiteUsers) — inside a Promise.all over every requested goal.

For a website with N goals, loading the goals dashboard issues up to 2×N concurrent ClickHouse round-trips on every request. A website with 20 goals fires ~40 queries where the actual underlying data (unique visitor counts scoped by date range) could largely be answered by a small, fixed number of queries.

Root cause

Each goal independently computes:

  1. getTotalWebsiteUsers(websiteId, effectiveStartDate, endDate, combinedFilters) — the entrant denominator
  2. processGoalConversionCount(step, combinedFilters, params) — the completion count for that goal's single step

Most goals on a given website share the same date range and carry no goal-specific filters, so steps 1 and 2 above are frequently computing the same denominator and issuing structurally identical completion queries that differ only in which single event/pageview they're matching — exactly the shape ClickHouse can answer in one query via GROUP BY.

Proposed fix

Group goals by (a) whether they carry any filters (request-level or goal-level) and (b) their effective start date:

  • Goals with no filters at all in the same date bucket can be counted in a single batched query — one shared getTotalWebsiteUsers call plus one query that matches all their step conditions and returns per-goal completion counts via GROUP BY, instead of N×2 queries.
  • Goals with filters keep the existing one-query-per-goal path unchanged, since the shared event-stream query builder (buildIdentifiedEventStream) only threads filter conditions through the step at array index 0 (correct for its actual funnel-entry-filter use case) — batching filtered goals together would silently apply another goal's filter to the wrong goal.

This bounds the fix to the case where it's provably correct without touching the shared funnel query builder, while still collapsing the common case (dashboards with mostly unfiltered goals) from O(N) queries to O(distinct date ranges).

Will open a PR shortly with the batched implementation, a regression test asserting the query count, and no change in output for existing behavior.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions