Update scikit-learn to v1.4

.gitlab-ci.yml

@@ -4,7 +4,7 @@ stages:
   - test
 
 variables:
-  PYVERSION: "3.8"
+  PYVERSION: "3.10"
   BINDIR: "/root/sklldev/bin"
   MPLBACKEND: "Agg"
   LOGCAPTURE_LEVEL: "WARNING"

.pre-commit-config.yaml

@@ -3,7 +3,7 @@
 
 repos:
   - repo: https://github.com/pre-commit/pre-commit-hooks
-    rev: v4.4.0
+    rev: v4.5.0
     hooks:
       - id: trailing-whitespace
       - id: end-of-file-fixer
@@ -14,24 +14,24 @@ repos:
       - id: check-json
       - id: debug-statements
   - repo: https://github.com/compilerla/conventional-pre-commit
-    rev: 'v2.2.0'
+    rev: 'v3.1.0'
     hooks:
       - id: conventional-pre-commit
         stages: [commit-msg]
   - repo: https://github.com/ikamensh/flynt/
-    rev: '0.78'
+    rev: '1.0.1'
     hooks:
       - id: flynt
   - repo: https://github.com/psf/black
-    rev: 23.3.0
+    rev: 24.2.0
     hooks:
       - id: black
         args: [--line-length=100]
   - repo: https://github.com/charliermarsh/ruff-pre-commit
-    rev: 'v0.0.264'
+    rev: 'v0.2.1'
     hooks:
       - id: ruff
-        args: [--line-length=100, --select, "D,E,F,I", --ignore, "D212", --per-file-ignores, "tests/test*.py:D102,tests/test*.py:D103,tests/test_input.py:E501,skll/data/featureset.py:E501,skll/learner/__init__.py:E501,skll/learner/voting.py:E501,skll/learner/utils.py:E501"]
+        args: [--line-length=100, --select, "D,E,F,I", --ignore, "D212", --per-file-ignores, "tests/test*.py:D,tests/test_input.py:E501,skll/data/featureset.py:E501,skll/learner/__init__.py:E501,skll/learner/voting.py:E501,skll/learner/utils.py:E501"]
   - repo: https://github.com/pre-commit/mirrors-mypy
     rev: 'v1.8.0'
     hooks:

doc/custom_metrics.rst

@@ -12,15 +12,27 @@ Writing Custom Metric Functions
 
 First, let's look at how to write valid custom metric functions. A valid custom metric function
 must take two array-like positional arguments: the first being the true labels or scores, and the
-second being the predicted labels or scores. This function can also take three optional keyword arguments:
+second being the predicted labels or scores. This function can also take two optional keyword arguments:
 
 1. ``greater_is_better``: a boolean keyword argument that indicates whether a higher value of the metric indicates better performance (``True``) or vice versa (``False``). The default value is ``True``.
-2. ``needs_proba``: a boolean keyword argument that indicates whether the metric function requires probability estimates. The default value is ``False``.
-3. ``needs_threshold``: a boolean keyword argument that indicates whether the metric function takes a continuous decision certainty. The default value is ``False``.
+
+2. ``response_method`` : a string keyword argument that specifies the response method to use to get predictions from an estimator. Possible values are:
+
+   - ``"predict"`` : uses estimator's `predict() <https://scikit-learn.org/stable/glossary.html#term-predict>`__ method to get class labels
+   - ``"predict_proba"`` : uses estimator's `predict_proba() <https://scikit-learn.org/stable/glossary.html#term-predict_proba>`__ method to get class probabilities
+   - ``"decision_function"`` : uses estimator's `decision_function() <https://scikit-learn.org/stable/glossary.html#term-decision_function>`__ method to get continuous decision function values
+   - If the value is a list or tuple of the above strings, it indicates that the scorer should use the first method in the list which is implemented by the estimator.
+   - If the value is ``None``, it is the same as ``"predict"``.
+
+   The default value for ``response_method`` is ``None``.
 
 Note that these keyword arguments are identical to the keyword arguments for the `sklearn.metrics.make_scorer() <https://scikit-learn.org/stable/modules/generated/sklearn.metrics.make_scorer.html#sklearn.metrics.make_scorer>`_ function and serve the same purpose.
 
-In short, custom metric functions take two required positional arguments (order matters) and three optional keyword arguments. Here's a simple example of a custom metric function: F\ :sub:`β` with β=0.75 defined in a file called ``custom.py``.
+.. important::
+
+   Previous versions of SKLL offered the ``needs_proba`` and ``needs_threshold`` keyword arguments for custom metrics but these are now deprecated in scikit-learn and replaced by the ``response_method`` keyword argument. To replicate the behavior of ``needs_proba=True``, use ``response_method="predict_proba"`` instead and to replicate ``needs_threshold=True``, use ``response_method=("decision_function", "predict_proba")`` instead.
+
+In short, custom metric functions take two required positional arguments (order matters) and two optional keyword arguments. Here's a simple example of a custom metric function: F\ :sub:`β` with β=0.75 defined in a file called ``custom.py``.
 
 .. code-block:: python
   :caption: custom.py
@@ -30,7 +42,6 @@ In short, custom metric functions take two required positional arguments (order
   def f075(y_true, y_pred):
       return fbeta_score(y_true, y_pred, beta=0.75)
 
-
 Obviously, you may write much more complex functions that aren't directly
 available in scikit-learn. Once you have written your metric function, the next
 step is to use it in your SKLL experiment.

requirements.txt

@@ -6,7 +6,7 @@ numpy
 pandas
 pre-commit
 ruamel.yaml
-scikit-learn>=1.3.0,<1.4.0
+scikit-learn>=1.4.0,<1.5.0
 scipy
 seaborn
 sphinx_rtd_theme==1.2.0

skll/experiments/output.py

@@ -12,6 +12,7 @@
 
 import csv
 import json
+import logging
 import math
 import sys
 from collections import defaultdict
@@ -26,7 +27,7 @@
 from ruamel.yaml import YAML
 
 from skll.types import FoldMapping, PathOrStr
-from skll.utils.logging import get_skll_logger
+from skll.utils.logging import MatplotlibCategoryFilter, get_skll_logger
 
 # Turn off interactive plotting for matplotlib
 plt.ioff()
@@ -143,12 +144,19 @@ def _generate_learning_curve_score_plots(
                 legend_out=False,
             )
             train_color, test_color = sns.color_palette(palette="Set1", n_colors=2)
+
+            # create a filter to hide unnecessary matplotlib.category warnings
+            filter = MatplotlibCategoryFilter()
+            logging.getLogger("matplotlib.category").addFilter(filter)
+
+            # map the FacetGrid to the data
             g = g.map_dataframe(
                 sns.pointplot,
                 x="training_set_size",
                 y="value",
                 hue="variable",
-                scale=0.5,
+                markersize=2,
+                linewidth=0.8,
                 errorbar=None,
                 palette={"train_score_mean": train_color, "test_score_mean": test_color},
             )
@@ -209,7 +217,7 @@ def _generate_learning_curve_score_plots(
                                 ncol=1,
                                 frameon=True,
                             )
-            g.fig.tight_layout(w_pad=1)
+            g.figure.tight_layout(w_pad=1)
             plot_file_path = output_dir / f"{experiment_name}_{fs_name}.png"
             plt.savefig(plot_file_path, dpi=300)
             # explicitly close figure to save memory
@@ -274,13 +282,16 @@ def _generate_learning_curve_time_plots(
                 sharey=True,
                 legend_out=False,
             )
+
             g = g.map_dataframe(
                 sns.pointplot,
                 x="training_set_size",
                 y="value",
                 hue="variable",
-                scale=0.5,
+                markersize=2,
+                linewidth=0.8,
                 errorbar=None,
+                palette="dark:#1f77b4",
             )
             # compute the upper and lower
             for ax in g.axes.flat:
@@ -303,7 +314,7 @@ def _generate_learning_curve_time_plots(
                     alpha=0.1,
                 )
 
-            g.fig.tight_layout(w_pad=1)
+            g.figure.tight_layout(w_pad=1)
             plot_file_path = output_dir / f"{experiment_name}_{fs_name}_times.png"
             plt.savefig(plot_file_path, dpi=300)
 

skll/learner/__init__.py

@@ -252,10 +252,12 @@ def __init__(
                     "produce probabilities, results will not be exactly "
                     "replicable when using SVC and probability mode."
                 )
+        elif issubclass(self._model_type, AdaBoostClassifier):
+            self._model_kwargs["algorithm"] = "SAMME"
+            self._model_kwargs["n_estimators"] = 500
         elif issubclass(
             self._model_type,
             (
-                AdaBoostClassifier,
                 AdaBoostRegressor,
                 BaggingClassifier,
                 BaggingRegressor,
@@ -268,6 +270,8 @@ def __init__(
             self._model_kwargs["n_estimators"] = 500
         elif issubclass(self._model_type, DummyClassifier):
             self._model_kwargs["strategy"] = "prior"
+        elif issubclass(self._model_type, (LinearSVC, LinearSVR)):
+            self._model_kwargs["dual"] = "auto"
         elif issubclass(self._model_type, SVR):
             self._model_kwargs["cache_size"] = 1000
             self._model_kwargs["gamma"] = "scale"
@@ -950,7 +954,7 @@ def train(
                 metrics_module = import_module("skll.metrics")
                 metric_func = getattr(metrics_module, "correlation")
                 _CUSTOM_METRICS[new_grid_objective] = make_scorer(
-                    metric_func, corr_type=grid_objective, needs_proba=True
+                    metric_func, corr_type=grid_objective, response_method="predict_proba"
                 )
                 grid_objective = new_grid_objective
 

skll/metrics.py

@@ -298,7 +298,7 @@ def register_custom_metric(custom_metric_path: PathOrStr, custom_metric_name: st
     # extract any "special" keyword arguments from the metric function
     metric_func_parameters = signature(metric_func).parameters
     make_scorer_kwargs = {}
-    for make_scorer_kwarg in ["greater_is_better", "needs_proba", "needs_threshold"]:
+    for make_scorer_kwarg in ["greater_is_better", "response_method"]:
         if make_scorer_kwarg in metric_func_parameters:
             parameter = metric_func_parameters.get(make_scorer_kwarg)
             if parameter is not None:

skll/utils/logging.py

@@ -16,6 +16,36 @@
 SKLEARN_WARNINGS_RE = re.compile(re.escape(f"{sep}sklearn{sep}"))
 
 
+class MatplotlibCategoryFilter(logging.Filter):
+    """
+    Class to filter out specific log records from `matplotlib.category`.
+
+    This is useful when generating learning curves which generates unnecessary
+    log records from `matplotlib.category`. For more details, see this issue:
+    https://github.com/matplotlib/matplotlib/issues/23422
+    """
+
+    def filter(self, record):
+        """
+        Implement the filter method.
+
+        Parameters
+        ----------
+        record : logging.LogRecord
+            The log record to be filtered.
+        """
+        # Check if the log record is from matplotlib.category and contains the specific message
+        if (
+            record.name == "matplotlib.category"
+            and "Using categorical units to plot a list of strings" in record.msg
+        ):
+            # filter out this record
+            return False
+
+        # allow other records through
+        return True
+
+
 def send_sklearn_warnings_to_logger(
     logger, message, category, filename, lineno, file=None, line=None
 ):

tests/configs/test_send_warnings_to_log.template.cfg

@@ -4,7 +4,7 @@ task=cross_validate
 
 [Input]
 featuresets=[["test_send_warnings_to_log"]]
-learners=["LinearSVC"]
+learners=["DummyClassifier"]
 suffix=.jsonlines
 num_cv_folds=2
 

tests/other/custom_metrics.py

@@ -1,3 +1,4 @@
+"""Custom metrics for testing purposes."""
 from sklearn.metrics import (
     average_precision_score,
     f1_score,
@@ -8,31 +9,31 @@
 )
 
 
-def f075_macro(y_true, y_pred):
+def f075_macro(y_true, y_pred):  # noqa: D103
     return fbeta_score(y_true, y_pred, beta=0.75, average="macro")
 
 
-def ratio_of_ones(y_true, y_pred):
+def ratio_of_ones(y_true, y_pred):  # noqa: D103
     true_ones = [label for label in y_true if label == 1]
     pred_ones = [label for label in y_pred if label == 1]
     return len(pred_ones) / (len(true_ones) + len(pred_ones))
 
 
-def r2(y_true, y_pred):
+def r2(y_true, y_pred):  # noqa: D103
     return r2_score(y_true, y_pred)
 
 
-def one_minus_precision(y_true, y_pred, greater_is_better=False):
+def one_minus_precision(y_true, y_pred, greater_is_better=False):  # noqa: D103
     return 1 - precision_score(y_true, y_pred, average="binary")
 
 
-def one_minus_f1_macro(y_true, y_pred, greater_is_better=False):
+def one_minus_f1_macro(y_true, y_pred, greater_is_better=False):  # noqa: D103
     return 1 - f1_score(y_true, y_pred, average="macro")
 
 
-def fake_prob_metric(y_true, y_pred, needs_proba=True):
+def fake_prob_metric(y_true, y_pred, response_method="predict_proba"):  # noqa: D103
     return average_precision_score(y_true, y_pred)
 
 
-def fake_prob_metric_multiclass(y_true, y_pred, needs_proba=True):
+def fake_prob_metric_multiclass(y_true, y_pred, response_method="predict_proba"):  # noqa: D103
     return roc_auc_score(y_true, y_pred, average="macro", multi_class="ovo")

tests/test_classification.py

@@ -542,7 +542,7 @@ def test_sparse_predict(self):  # noqa: D103
                 (0.45, 0.52),
                 (0.52, 0.5),
                 (0.48, 0.5),
-                (0.49, 0.5),
+                (0.5, 0.5),
                 (0.54, 0.5),
                 (0.43, 0),
                 (0.53, 0.57),
@@ -814,8 +814,8 @@ def check_adaboost_predict(self, base_estimator, algorithm, expected_score):
     def test_adaboost_predict(self):  # noqa: D103
         for base_estimator_name, algorithm, expected_score in zip(
             ["MultinomialNB", "DecisionTreeClassifier", "SGDClassifier", "SVC"],
-            ["SAMME.R", "SAMME.R", "SAMME", "SAMME"],
-            [0.46, 0.52, 0.46, 0.5],
+            ["SAMME", "SAMME", "SAMME", "SAMME"],
+            [0.49, 0.52, 0.46, 0.5],
         ):
             yield self.check_adaboost_predict, base_estimator_name, algorithm, expected_score
 

tests/test_logging_utils.py

@@ -4,6 +4,7 @@
 :author: Nitin Madnani (nmadnani@ets.org)
 """
 
+import logging
 import re
 import sys
 import unittest
@@ -14,6 +15,7 @@
 from sklearn.metrics import roc_curve
 
 from skll.utils.logging import (
+    MatplotlibCategoryFilter,
     close_and_remove_logger_handlers,
     get_skll_logger,
     orig_showwarning,
@@ -135,3 +137,44 @@ def test_close_and_remove_logger_handlers(self):
         LOGGERS.append(logger)
         close_and_remove_logger_handlers(logger)
         assert not logger.handlers
+
+
+class TestMatplotlibCategoryFilter(unittest.TestCase):
+    """Test class for the MatplotlibCategoryFilter."""
+
+    def setUp(self):
+        # set up the logger and add the custom filter
+        self.logger = logging.getLogger("matplotlib.category")
+        self.logger.setLevel(logging.INFO)
+        self.filter = MatplotlibCategoryFilter()
+        self.logger.addFilter(self.filter)
+
+        # set up a logging handler to capture the logs
+        self.log_capture = logging.handlers.MemoryHandler(capacity=10, target=None)
+        self.logger.addHandler(self.log_capture)
+
+    def tearDown(self):
+        # remove the filter and handler
+        self.logger.removeFilter(self.filter)
+        self.logger.removeHandler(self.log_capture)
+
+    def test_filtered_message(self):
+        # log a message that should be filtered
+        self.logger.info(
+            "Using categorical units to plot a list of strings that are all "
+            "parsable as floats or dates."
+        )
+
+        # Check that the message was filtered and not captured
+        self.assertEqual(len(self.log_capture.buffer), 0)
+
+    def test_allowed_message(self):
+        # log a message that should not be filtered
+        self.logger.info("This is a test message that should not be filtered.")
+
+        # check that the message was not filtered and captured
+        self.assertEqual(len(self.log_capture.buffer), 1)
+        self.assertEqual(
+            self.log_capture.buffer[0].getMessage(),
+            "This is a test message that should not be filtered.",
+        )