Skip to content

Native metrics do not always get mapped to the correct Spark nodes #1109

Closed

Description

@andygrove

Describe the bug

CometNativeExec builds a tree of CometMetric nodes containing SQLMetrics for the Spark plan operators. This structure is passed into the Native.createPlan method.

During native execution, calls to update_metrics will recurse through the native ExecutionPlan tree and through the CometMetric tree simultaneously and copy native metrics to Spark metrics.

This approach assumes a 1:1 mapping between the Spark plan and the native plan. In some cases, this assumption is not correct and we therefore drop metrics and the Spark UI does not show metrics for some nodes.

One example is for aggregate queries, where the native planner sometimes inserts an additional projection.

Another example is that we insert CopyExec nodes around the inputs to aggregate, sort, and join operators.

Steps to reproduce

No response

Expected behavior

No response

Additional context

No response

Activity

  1. self-assigned this
    on Nov 21, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions