Add VotingClassifier and VotingRegressor ensembles.

[desilinguist]

These would be great for baseline ensembles.

[desilinguist]

This is actually a bit more complicated because it requires passing in the base estimators whose predictions will be voted on. This requires a bit more thinking,.

[desilinguist]

Here's how I am planning to implement this.

Given that the main Learner class in SKLL is already quite complex, I am a little reluctant to tack further functionality onto it. Therefore, I propose that we create a new class that will inherit from Learner: VotingLearner which will live in learner/voting.py.

We would instantiate a voting learner as follows via the API:

from skll.learner import VotingLearner

VotingLearner(type="classifier",
              estimators=['LogisticRegression', 'SVC'],
              voting="hard",
              custom_learner_path=None,
              probability=False,
              feature_scaling='none',
              min_feature_count=1,
              pos_label_str=None,
              sampler_list=None,
              sampler_kwargs_list=None,
              model_kwargs_list=None,
              logger=None)

• All fields other than type and estimators are optional and are only there to pass in the other values to the underlying Learner instances that will be created.
• The tricky bit here is that after you instantiate a VotingClassifier with the estimators and then you call fit() on it, it basically creates clones of the underlying estimators and fits them too. However, since we do our own pre-processing in SKLL as part of the training (scaling, feature selection, sampling etc.) that varies by type of learner (e.g., we don't scale for MultinomialNB learners), this will be a real headache. How do we make sure that every learner is processing the training data X, y passed through VotingClassifier.fit(X, y) correctly? What saves us here is the pipeline attribute for each trained learner. We can just use this entire pipeline as the estimator. This makes sure that everything carries through correctly.
• As part of this specific instantiation call, we will create two new Learner instances, one for LogisticRegression and one for SVC in the __init__() method and save them in an attribute, say, self.estimators_. We will also save the names of the estimators under self.estimator_names. We also save either VotingClassifier or VotingRegressor (depending on the type value) under self._model_type just like we do for regular learners.
• We will override the train() method to call train() on each of the learners in self.estimators_ and then fit the VotingClassifier or VotingRegressor using the already trained estimators and save that under self._model.
• We will override the predict() method to just predict() on the fit VotingClassifier or VotingRegressor which should automatically take care of calling predict() on the underlying estimators.
• We will override the cross_validate() method to call cross_validate() on the underlying estmators, get their cross-validated predictions, and then just call self.predict().
• The evaluate() and learning_curve() methods should just work since they rely on train(), cross_validate(), and predict().

A configuration file with VotingLearner would look something like this:

[General]
experiment_name = Titanic_Voting
task = evaluate

[Input]
train_directory = train
test_directory = dev
fixed_parameters = [{}, {"estimator_names: ["LogisticRegression", "SVC"], estimator_fixed_parameters=[{}, {}], estimator_param_grids=[{}, {}]}]
featuresets = [["family.csv", "misc.csv", "socioeconomic.csv", "vitals.csv"]]
learners = ["RandomForestClassifier", "VotingClassifier"]
label_col = Survived
id_col = PassengerId

[Tuning]
grid_search = true
objectives = ['accuracy']

[Output]
log = output
results = output
predictions = output
models = output

We just need to make it such that some fixed_parameters are required for a VotingClassifier/VotingRegressor which will help us create the right VotingLearner instance underlyingly.