AutoML experiments in non declarative style not working

System Information (please complete the following information):

OS & Version: Windows 11
ML.NET Version: 2.0
.NET Version: .NET6

Describe the bug
Running the "old" AutoML experiments does not work for all trainers. Using CreateMulticlassClassificationExperiment or CreateRegressionExperiment instead of the new declarative style will result in an exception (possibly due to custom schemas).

An exception will be raised in the new SweepablePipeline:

System.NullReferenceException: Object reference not set to an instance of an object.
   at Microsoft.ML.AutoML.SweepablePipeline..ctor(Dictionary`2 estimators, Entity schema, String currentSchema)
   at Microsoft.ML.AutoML.SweepablePipeline.AppendEntity(Boolean allowSkip, Entity entity)
   at Microsoft.ML.AutoML.AutoCatalog.MultiClassification(String labelColumnName, String featureColumnName, String exampleWeightColumnName, Boolean useFastForest, Boolean useLgbm, Boolean useFastTree, Boolean useLbfgs, Boolean useSdca, FastTreeOption fastTreeOption, LgbmOption lgbmOption, FastForestOption fastForestOption, LbfgsOption lbfgsOption, SdcaOption sdcaOption, SearchSpace`1 fastTreeSearchSpace, SearchSpace`1 lgbmSearchSpace, SearchSpace`1 fastForestSearchSpace, SearchSpace`1 lbfgsSearchSpace, SearchSpace`1 sdcaSearchSpace)
   at Microsoft.ML.AutoML.MulticlassClassificationExperiment.CreateMulticlassClassificationPipeline(IDataView trainData, ColumnInformation columnInformation, IEstimator`1 preFeaturizer)
   at Microsoft.ML.AutoML.MulticlassClassificationExperiment.Execute(IDataView trainData, ColumnInformation columnInformation, IEstimator`1 preFeaturizer, IProgress`1 progressHandler)
   at Microsoft.ML.AutoML.MulticlassClassificationExperiment.Execute(IDataView trainData, String labelColumnName, String samplingKeyColumn, IEstimator`1 preFeaturizer, IProgress`1 progressHandler)

To Reproduce

var experimentSettings = new MulticlassExperimentSettings
{
  MaxExperimentTimeInSeconds = 30,
  OptimizingMetric = MulticlassClassificationMetric.MacroAccuracy
};
experimentSettings.Trainers.Clear();
experimentSettings.Trainers.Add(MulticlassClassificationTrainer.LbfgsMaximumEntropy);
ctx.Auto().CreateMulticlassClassificationExperiment(experimentSettings).Execute(trainDv);

Where the schema is of a custom type:

var schemaDef = SchemaDefinition.Create(typeof(ModelInput));
schemaDef["Features"].ColumnType = new VectorDataViewType(NumberDataViewType.Single, numberOfFeatures);
schemaDef.Remove(schemaDef["LabelFeaturized"]); // removed when not applicable

public class ModelInput
{
  public uint Label;
  public float LabelFeaturized;
  public float[] Features;
}

Expected behavior
No regression expected for CreateMulticlassClassificationExperiment and CreateRegressionExperiment.

Thanks for reporting this bug. It's because LbfgsMaximumEntropy is not used when constructing sweepable pipeline for multiclass classification (

machinelearning/src/Microsoft.ML.AutoML/API/MulticlassClassificationExperiment.cs

Line 314 in 17c061a

    
           private SweepablePipeline CreateMulticlassClassificationPipeline(IDataView trainData, ColumnInformation columnInformation, IEstimator<ITransformer> preFeaturizer = null)

)

We'll push a fix for this bug, in the meanwhile, you can add other trainer (like lightGbm) as a work-around

Thanks for reporting this bug. It's because LbfgsMaximumEntropy is not used when constructing sweepable pipeline for multiclass classification (

machinelearning/src/Microsoft.ML.AutoML/API/MulticlassClassificationExperiment.cs

Line 314 in 17c061a

private SweepablePipeline CreateMulticlassClassificationPipeline(IDataView trainData, ColumnInformation columnInformation, IEstimator<ITransformer> preFeaturizer = null)

)

We'll push a fix for this bug, in the meanwhile, you can add other trainer (like lightGbm) as a work-around

Thank you for your answer! We will downgrade until the fix is deployed.

Any chance of getting this fix rolled out before ML.NET 3.0 as an update to ML.NET 2.0? Projects depending on this would otherwise need to stay at 0.7.1/ 0.19.1 :(

@luisquintanilla

@luisquintanilla how does the suggestion by @zeroskyx sound for a servicing release?

I believe it should be in a servicing release as it will produce a runtime exception when simply upgrading and using a previously functional workflow (without any syntax or compilation warnings / errors).

Possibly related to #6529

fix #6446 (#6447)

Navigation Menu

Repository navigation

AutoML experiments in non declarative style not working #6446

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

AutoML experiments in non declarative style not working #6446

Description

Activity

LittleLittleCloud commented on Nov 10, 2022

thoron commented on Nov 11, 2022

zeroskyx commented on Dec 3, 2022

michaelgsharp commented on Dec 5, 2022

thoron commented on Dec 6, 2022

zeroskyx commented on Dec 23, 2022

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions