Repository navigation
dimension_anomalies is failing for a given dimension when it shouldn't #2172
Description
Describe the bug
I have a table that has data ingested into it daily, all rows using the same ingestion timestamp (for the sake of this issue, let's call it ts). I'm trying to implement an elementary test based on a given dimension called row_type, so that I can evaluate if the volume for each dimension is still accurate or if it has drifted with time.
I found thedimension_anomalies test, but when I tried to use it, it returned inconsistent training statistics when using timestamp_column=ts and dimension=row_type. In my case specifically, latest_metric_value in table elementary_db.metrics_anomaly_score reflects correctly what I have in the source if I group by row_type, however, the reported training_avg is far below that dimension's historical values.
The main thing is that, for this particular row_type dimension value X, it has 30x more data than the reported avg and historically, doesn't have any anomaly as values have been pretty aligned since the table exists, but still, it's being reported as an anomaly, consequently triggering an error. Could you please check?
P.S.: I have 40+ distinct row_type and some of them have just a few records in comparison with the main ones, with millions of rows every single day. I believe elementary dimension_anomalies should take the avg grouped by dimension, but if that's not the case by design, I'm not sure how this can be useful. Not sure if it's due to this issue reported and closed in the past, or if it's something else.
To Reproduce
N/A
Expected behavior
Assuming I have dimension_anomalies in my sources.yml set as:
config:
elementary:
timestamp_column: ts
...
- elementary.dimension_anomalies:
arguments:
dimensions:
- row_type
anomaly_direction: both
time_bucket:
period: day
count: 1
If you have a dimension with distinct values X, Y and Z, and their counts, in day1, day2, day3 and day4 are:
| D1 | D2 | D3 | D4 | |
|---|---|---|---|---|
| X | 1000 | 1001 | 1010 | 1002 |
| Y | 200 | 200 | 201 | 7000 |
| Z | 10 | 10 | 9 | 10 |
Assuming anomaly is already trained, and it's configured to trigger in both directions, we should expect anomaly alert in D4 for dimension value Y because it has deviated from the Y historic avg/std or whatever algorithm it uses.
But in my case, X is being called anomalous since D1.
Screenshots
N/A
Environment (please complete the following information):
- Elementary CLI (edr) version: [e.g. 0.5.3], can be found by running
pip show elementary-data - Elementary dbt package version: [e.g. 0.4.1], can be found in
packages.ymlfile: 0.20.1 (tested with 0.23.0, no success) - dbt version you're using [e.g. 1.8.1]: 1.11.2
- Data warehouse [e.g. snowflake]: AWS Athena
- Infrastructure details (e.g. operating system, prod / dev / staging, deployment infra, CI system, etc)
Additional context
Would you be willing to contribute a fix for this issue?
Activity
github-actions commentedon May 24, 2026
This issue is stale because it has been open for too long with no activity.
If you would like the issue to remain open, please remove the stale label or leave a comment.
daykodotexe commentedon Jun 24, 2026
I'm looking into this and plan to submit a fix. Can confirm the training_avg grouping behavior looks like the root cause — will share findings shortly
sravankumarkunadi commentedon Jul 27, 2026
@daykodotexe — did you get anywhere with this? I've dug in and opened a PR for part of it, but I don't want to duplicate your work, so happy to defer if you already have something in flight.
Sharing what I found, since it looks like there are two separate things in this report.
1. The
training_avgyou're seeing inmetrics_anomaly_score— confirmed bug, PR open@cabral1888's observation that "
latest_metric_valuereflects the source correctly when grouped byrow_type, buttraining_avgis far below that dimension's historical values" is exactly the bug from #1729, and it's still unfixed onmaster.In
models/edr/data_monitoring/anomaly_detection/metrics_anomaly_score.sql, all five window functions partition bymetric_name, full_table_name, column_name—dimensionanddimension_valueare selected and in thegroup by, but missing from the partitions. So every dimension value gets pooled into one training set.That matches the reported symptom precisely. With the counts in the issue, the pooled average is ~400, so
X(flat at ~1000 for its whole history) sits far above it and reads anomalous on every bucket, while the realYspike at D4 gets diluted. The distortion scales with how uneven the dimension values are — which fits "40+ distinctrow_type, some with a few records and some with millions."Fix is in elementary-data/dbt-data-reliability#1042.
2. The test failure itself — probably a different cause
I want to flag this rather than let the PR imply more than it delivers.
The test doesn't read
training_avgfrom that view. It comes frommacros/edr/data_monitoring/anomaly_detection/get_anomaly_scores_query.sql, which already partitions bymetric_name, full_table_name, column_name, dimension, dimension_value(partition_by_keys, line 62) — and has done since 2022 (c38a70eb). So that path was already correct in both 0.20.1 and 0.23.0.Which means PR elementary-data/dbt-data-reliability#1042 fixes the numbers you're reading in
elementary_db.metrics_anomaly_score, but likely won't stop the test from failing. Either the failure has another cause, or the view is what led to the diagnosis.@cabral1888, a couple of things that would help narrow it down:
ts. Does anything change if you compare against a column with intra-day spread? Curious whether zero spread interacts with the bucket matching incomplete_buckets_cte.training_set_size,training_startandtraining_endfromanomaly_scores(the test table, not themetrics_anomaly_scoreview) for theXrows that are failing? If the training window is short or starts in the wrong place, that points somewhere other than the partitioning.I'd suggest keeping this open after elementary-data/dbt-data-reliability#1042 merges, since it only covers the first half.
github-actions commentedon Sep 26, 2026
This issue is stale because it has been open for too long with no activity.
If you would like the issue to remain open, please remove the stale label or leave a comment.
EladBarkay commentedon Oct 5, 2026
Thanks for the detailed report, and thanks @sravankumarkunadi for the investigation.
We looked into this and found two separate things:
metrics_anomaly_scoreview was wrong. It computedtraining_avgacross all dimension values together instead of per value, so a large, stable value like yourXlooked anomalous there. This is fixed in fix: partition metrics_anomaly_score training stats by dimension dbt-data-reliability#1042, released in 0.26.0.dimension_anomaliestest itself doesn't use that view. It computes its training stats per dimension value. We reproduced your setup (one ingestion timestamp per day, 40+row_typevalues with very uneven volumes, your exact test config) on packages 0.20.1, 0.23.0 and 0.26.0. In all three the test passed and didn't flag the large value, while the old view still showed the wrongtraining_avg.So the numbers that pointed at the problem came from the buggy view, and we couldn't reproduce the test failure itself. Could you upgrade to 0.26.0 and rerun? If the test still fails, please share these fields for the flagged
Xrows, from the test's own results (test_result_rows/ the test's failing rows), not the view:bucket_end,value,training_avg,training_stddev,training_set_size,training_start,training_end. That would tell us whether something specific to Athena is going on.We'll close this in two weeks if we don't hear back. Feel free to reopen anytime.