[TeddyCr]

Summary

CostAnalysisWorkflow and WebAnalyticsWorkflow recreated processor, sink, and aggregator objects with constant step names on every per-service or per-day iteration, so the plain put in WorkflowStats.updateWorkflowStepStats overwrote each previous iteration and the persisted AppRunRecord stats reflected only the last service or day. Following the approach recommended in issue #7403, both workflows now format each constant-named step as a base name qualified with the per-service or per-day source name, producing unique step keys per iteration. WorkflowStats is intentionally unchanged because its replace semantics are relied on by DataAssetsWorkflow's cumulative per-entity-type and [Enricher]-prefixed step keys, and DataInsightsApp is intentionally unchanged because updateStats already computes job-level totals as a sum over every entity-stats entry, which now aggregates all services and all days. Reviewers should look first at the step-name changes in CostAnalysisWorkflow.java and WebAnalyticsWorkflow.java; the new WorkflowStatsBatchAggregationTest covers per-step key uniqueness and job-level aggregation.

Implementer's note: Per-service and per-day step keys are now unique in the cost-analysis and web-analytics workflows, a new WorkflowStatsBatchAggregationTest passes, and the planned spotless check passes.

Plan

0. Worktree prerequisite: the OpenMetadata submodule must be checked out in the implementation worktree
   (git submodule update --init --remote OpenMetadata). All edits go in the submodule working tree, per
   the task card (issue UI : Explore page data loading issue #7403 points at these submodule paths; the plan worktree's submodule is empty).
1. CostAnalysisWorkflow.java — thread the per-service qualifier through the per-service loop and
   suffix every constant-named step:
   • The loop already has source whose name (source.getName(), e.g. [CostAnalysisWorkflow] <FQN>) is
     unique per service; sources already use it as their step key (CostAnalysisWorkflow.java:210) — leave
     that unchanged.
   • Give processRawCostAnalysisReportData(List, Map) and processAggregatedCostAnalysisReportData(Map, Map)
     an extra String stepKeyQualifier parameter; pass source.getName() from the loop (call sites at
     CostAnalysisWorkflow.java:222 and :231).
   • Inside those two methods, build each step name as String.format("%s (%s)", baseName, stepKeyQualifier)
     — apply to: raw-data processor (:242-246), raw-data sink (:266-270), aggregator (:292-293),
     aggregated-data processor (:315-319), aggregated-data sink (:339-343). Only the names change; stats
     collection and updateWorkflowStats (put + recompute totals, :390-403) stay as-is.
2. WebAnalyticsWorkflow.java — same treatment per backfill day:
   • Give processEntityViewData(Map, Map) and processUserActivityData(Map, Map, Map) an extra
     String stepKeyQualifier parameter; pass source.getName() (already unique per day — it embeds the
     date, PaginatedWebAnalyticEventDataSource.java:53-56) from the loop (:168, :177).
   • Suffix into: entity-view processor (:228-232), entity-view sink (:252-256), user-activity aggregator
     (:277-278), user-activity processor (:293-297), user-activity sink (:318-322).
   • updateWorkflowStats (:357-370) unchanged.
3. WorkflowStatsBatchAggregationTest.java (new) — plain JUnit unit test, no DB; package
   org.openmetadata.service.apps.bundles.insights.workflows. Drives the real workflow recording path so
   it fails (200 / 70) before the fix and passes (300 / 120) after:
   • Instantiate new CostAnalysisWorkflow(new CostAnalysisConfig().withEnabled(true), timestamp, 100, Optional.empty())
     (constructor is DB-free). Invoke the private processRawCostAnalysisReportData reflectively twice
     with qualifiers svc-a / svc-b and inputs of 100 / 200 raw report data rows, contextData carrying
     TimestampUtils.TIMESTAMP_KEY.
   • CreateReportDataProcessor.process is pure; ReportDataSink.write needs
     Entity.getEntityTimeSeriesRepository(...) — follow the established pattern in
     PaginatedWebAnalyticEventDataSourceReadFailureTest.java:57:
     mockStatic(Entity.class, CALLS_REAL_METHODS) returning Mockito.mock(ReportDataRepository.class)
     (its createNewRecord needs no stubbing).
   • Assert: the step-stats map

Verification

status: green (every planned check passed or was already seen green) · 3 checks: 3 skipped on evidence
- [skipped] mvn test -f pom.xml -pl openmetadata-service -Dtest=WorkflowStatsBatchAggregationTest -DfailIfNoTests=false  (seen green after the last change)
- [skipped] mvn test -f pom.xml -pl openmetadata-service -Dtest=DataInsightsAppTest+PaginatedWebAnalyticEventDataSourceReadFailureTest+WebAnalyticsUserActivityAggregatorTest -DfailIfNoTests=false  (seen green after the last change)
- [skipped] mvn spotless:check -f pom.xml -pl openmetadata-service  (seen green after the last change)

Review (glm): approve

• [low] openmetadata-service/src/test/java/org/openmetadata/service/apps/bundles/insights/workflows/WorkflowStatsBatchAggregationTest.java: The test invokes the private process methods reflectively with hand-picked qualifiers ("svc-a"/"svc-b", "day-1"/"day-2"), so it verifies the suffixing logic but never pins that the production loops actually pass a per-iteration-unique source.getName().
• [low] openmetadata-service/src/main/java/org/openmetadata/service/apps/bundles/insights/workflows/costAnalysis/CostAnalysisWorkflow.java: Suffixing with the full source display name produces redundant run-record keys like "[CostAnalysisWorkflow] Raw Cost Analysis Report Data Processor ([CostAnalysisWorkflow] )" (same pattern in WebAnalyticsWorkflow.java:343), since source names already carry the workflow prefix.
• [low] openmetadata-service/src/test/java/org/openmetadata/service/apps/bundles/insights/workflows/WorkflowStatsBatchAggregationTest.java: Only the raw-cost-analysis and entity-view paths are exercised; the equally-modified aggregator, aggregated processor/sink, and user-activity processor/sink/aggregator steps have no test coverage.

Files

.../costAnalysis/CostAnalysisWorkflow.java         |  30 +++--
 .../webAnalytics/WebAnalyticsWorkflow.java         |  29 +++--
 .../WorkflowStatsBatchAggregationTest.java         | 130 +++++++++++++++++++++
 3 files changed, 171 insertions(+), 18 deletions(-)

Closes #7403

Opened by taskr run 0147.

[gitar-bot]

Code Review ✅ Approved

🟡 Medium risk · Qualifying repeated workflow steps by service or day changes persisted AppRunRecord aggregation; non-unique qualifiers could leave execution totals undercounted.

Fixes the Data Insights run-record undercount in CostAnalysisWorkflow and WebAnalyticsWorkflow by suffixing constant step names with the per-service or per-day source name, so per-iteration stats are no longer overwritten. No issues found.

Verification

Rules No rules evaluated

Functional validation Not enabled

Set up

Auto-approval Not enabled

Set up

_{For more verification details, comment gitar display:verbose.}

Options

Display: compact → Showing compact verification results.

Comment with these commands to change the behavior for this request:

Compact

gitar display:verbose

_{Was this helpful? React with 👍 / 👎 | Powered by Gitar — free for open source}