Add W&B logging

[tamarl08]

This PR adds W&B logging for all task types.

Changes include:

• wandb logging for each task - see links and details below.
• using the return value of _classify_featureset directly (This part is not tested for the grid use case).
• exposing some parameters in the result dict so wandb logger can access them. this will affect the results.json file.

Examples:

train: currently only logs the configuration, model file path and train set size (in run summary).

predict: logs configuration, train and test sizes and predictions file

learning curve: logs config, plots and some raw data from the learning curve results (in run summary)

evaluate: logs config and all evaluation metrics and confusion matrix as a chart.

cross_validate: logs evaluation per fold + average of folds

TODO:

• update test data
• add tests of new functionality (?)
• revert changes in examples cfg files before merging

[desilinguist]

@tamarl08 this looks pretty elegant for the most part! I like that you added what you needed to the learner result dictionaries and then just use those for logging most of the time.

I made some minor suggestions based on the code itself but without testing what the various W&B outputs look like. I will do that next.

[desilinguist]

@tamarl08 any idea why gitlab is failing and Azure is passing?

[tamarl08]

Will check. I didn't expect anything to pass!

[desilinguist]

I am seeing things like this in my run:

I am also seeing charts with a single data point (e.g., the accuracy values etc.) which aren't useful. I wonder if we can tell W&B to create relevant charts from the summary file on the fly? Let's sit down next week and try to figure out what things are actually valuable to log and how to make a run appear useful right when someone opens it.

[tamarl08]

This is why I tried to log to summary instead, but I couldn't always get rid of these charts. I'll try some more and let's talk next week.

[pep8speaks]

Hello @tamarl08! Thanks for updating this PR.

• In the file tests/test_regression.py:

Line 452:64: E203 whitespace before ':'

Comment last updated at 2024-01-26 21:47:38 UTC

[tamarl08]

@desilinguist @mulhod @damien2012eng @Frost45
This is ready for review now. I changed the logging of evaluation/cv tasks so that no unneeded charts are logged.

See the updated evaluation logging here - look only at the most recent run.

Changes to tests are due to: a change I made to the job name/output file names; data added to the result dict; bug fixes in some tests.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (7c09d07) 95.33% compared to head (1dec165) 95.44%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #761      +/-   ##
==========================================
+ Coverage   95.33%   95.44%   +0.11%     
==========================================
  Files          30       30              
  Lines        3598     3688      +90     
==========================================
+ Hits         3430     3520      +90     
  Misses        168      168              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[desilinguist]

Looks really good! I ran some test experiments and W&B logging now looks much better than before. I made some minor suggestions. I am assuming that you are going to add documentation in a separate MR?

[tamarl08]

Thanks for the review @desilinguist! All suggestions applied. I did add a section in the docs, and also a comment about the output file names. Will add more examples later.

[desilinguist]

Just some minor tweaks for the documentation.

[mulhod]

Since I see others commenting on the docs, I will leave some comments here for now and continue my review after refreshing and/or waiting, etc.

[desilinguist]

LGTM!