Skip to content

Support multiple results comparison in benchmark scripts - #14196

Closed
Eason0729 wants to merge 4 commits into
apache:mainfrom
Eason0729:main
Closed

Eason0729 wants to merge 4 commits into
apache:mainfrom
Eason0729:main

Conversation

@Eason0729

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Close #13446

Rationale for this change

It would be useful to display the difference among multiple benchmark runs.

See issue.

What changes are included in this PR?

I modify bench.sh and compare.py to accept multiple arguments

Are these changes tested?

Are there any user-facing changes?

No.

@Eason0729

Copy link
Copy Markdown
Contributor Author

I spent some time trying to make it(compare.py) more readable, and I ended up with original code.
Let me know if there is a better way to do this.

Anyway, I thinks it ready for review.

@Eason0729
Eason0729 marked this pull request as ready for review January 22, 2025 02:53
@alamb

alamb commented Jan 23, 2025

Copy link
Copy Markdown
Contributor

Thanks @Eason0729 -- is there any chance you can post an example output after this change?

@Eason0729

Copy link
Copy Markdown
Contributor Author

Thanks @Eason0729 -- is there any chance you can post an example output after this change?

../bench.sh compare main main2 main3
Comparing main2 main3 to main
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓
┃ Query        ┃       main ┃      main2 ┃ Change on main2 ┃      main3 ┃ Change on main3 ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩
│ Qsort utf8   │ 25191.05ms │ 25191.05ms │       no change │ 25191.05ms │       no change │
│ Qsort int    │ 30908.16ms │ 30908.16ms │       no change │ 40908.16ms │    1.32x slower │
│ Qsort        │ 25652.09ms │ 25652.09ms │       no change │ 25652.09ms │       no change │
│ decimal      │            │            │                 │            │                 │
│ Qsort        │ 32701.67ms │ 32701.67ms │       no change │ 32701.67ms │       no change │
│ integer      │            │            │                 │            │                 │
│ tuple        │            │            │                 │            │                 │
│ Qsort utf8   │ 25588.63ms │ 25588.63ms │       no change │ 25588.63ms │       no change │
│ tuple        │            │            │                 │            │                 │
│ Qsort mixed  │ 29063.22ms │ 29063.22ms │       no change │ 29063.22ms │       no change │
│ tuple        │            │            │                 │            │                 │
└──────────────┴────────────┴────────────┴─────────────────┴────────────┴─────────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┓
┃ Benchmark Summary              ┃             ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━┩
│ Total Time (main)              │ 169104.82ms │
│ Average Time (main)            │  28184.14ms │
│ Total Time (main2)             │ 169104.82ms │
│ Average Time (main2)           │  28184.14ms │
│ Queries Faster (main2)         │           0 │
│ Queries Slower (main2)         │           0 │
│ Queries with No Change (main2) │           6 │
│ Total Time (main3)             │ 179104.82ms │
│ Average Time (main3)           │  29850.80ms │
│ Queries Faster (main3)         │           0 │
│ Queries Slower (main3)         │           1 │
│ Queries with No Change (main3) │           5 │
└────────────────────────────────┴─────────────┘

@github-actions

Copy link
Copy Markdown

Thank you for your contribution. Unfortunately, this pull request is stale because it has been open 60 days with no activity. Please remove the stale label or comment or this will be closed in 7 days.

@github-actions github-actions Bot added the Stale PR has not had any activity for some time label Mar 25, 2025
@github-actions github-actions Bot closed this Apr 9, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Stale PR has not had any activity for some time

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support multiple (>2) results comparison in benchmark scripts

2 participants