Stop using sklearn's private _scorer API for custom metrics in SKLL. - #751
Merged
Conversation
- Move all custom pre-defined metrics to `_PREDEFINED_CUSTOM_METRICS` dictionary in `metrics.py`. - Make a copy of the above dictionary as `CUSTOM _METRICS`. This dictionary will be the source of all custom metrics: pre-defined and user-definqed. - We need two dictionaries because at some points in the code, we need to be able to differentiate between pre-defined custom metrics and user-defined custom metrics. - Remove any use of `sklearn.metrics._scorer` private API and use `sklearn.metrics.get_scorer()` and `sklearn.metrics.get_scorer_names()` instead. - Use the actual metric function when using custom metrics instead of its name. This is the core change that makes everything work since sklearn validates callables but does not validate custom metric name strings.
desilinguist
requested review from
Frost45,
damien2012eng,
dblandan,
mulhod and
tamarl08
July 7, 2023 16:01
Codecov ReportPatch coverage:
Additional details and impacted files@@ Coverage Diff @@
## main #751 +/- ##
==========================================
+ Coverage 95.24% 95.30% +0.05%
==========================================
Files 29 29
Lines 3578 3576 -2
==========================================
Hits 3408 3408
+ Misses 170 168 -2
☔ View full report in Codecov by Sentry. |
Frost45
reviewed
Jul 11, 2023
Frost45
left a comment
Frost45
left a comment
Contributor
There was a problem hiding this comment.
I would like to suggest a couple of changes to the custom_metrics documentation:
- Fix indentation in custom.py. There are a few spaces at the start of every line.
- Change
logtologsin the config file on the same page. Otherwise, you get a KeyError.
Frost45
approved these changes
Jul 12, 2023
Frost45
left a comment
Frost45
left a comment
Contributor
There was a problem hiding this comment.
Looks great! Tested it out with multiple custom metrics.
Contributor
|
Tests passed for me just now. Will review shortly. |
mulhod
approved these changes
Jul 12, 2023
Co-authored-by: Matt Mulholland <mulhodm@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
_PREDEFINED_CUSTOM_METRICSdictionary inmetrics.py.CUSTOM _METRICS. This dictionary will be the source of all custom metrics: pre-defined and user-defined.sklearn.metrics._scorerprivate API and usesklearn.metrics.get_scorer()andsklearn.metrics.get_scorer_names()instead._SCORERSand fix a minor bug in another test related to scikit-learn v1.3.0.This PR closes #748 and closes #750.
To review this PR, please try to create some custom metrics for both the titanic and California examples and try to use them via both the API and the configuration file.