ENH: early stopping in models & example with RUMnet

[VincentAuriau]

No description provided.

[gemini-code-assist]

Summary of Changes

Hello @VincentAuriau, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request enhances the model training capabilities by integrating an early stopping mechanism into the base model, which helps prevent overfitting and optimizes training duration. It also refines the logging of validation metrics for clarity and provides a practical example of early stopping in action within the RUMnet model's notebook.

Highlights

• Early Stopping Implementation: Introduced early stopping logic within the fit method of the base model, allowing training to halt prematurely if self.stop_training is activated.
• Validation Loss Metric Renaming: Renamed the test_loss key to val_loss in the losses_history dictionary to standardize the naming convention for validation metrics.
• RUMnet Early Stopping Example: Added a new section to the RUMnet Jupyter notebook demonstrating how to configure and use tf.keras.callbacks.EarlyStopping with the model.
• Notebook Metadata Updates: Updated the Jupyter notebook's kernel display name to 'choice_learn' and the Python version to '3.12.11'.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces early stopping functionality to the base model and adds a corresponding example in the RUMnet notebook. The implementation in base_model.py correctly uses the self.stop_training flag set by Keras callbacks.

My review identifies a couple of issues:

• There's a bug where the early stopping check is missing for the training path where sample_weight is not used, which would cause early stopping to fail.
• The messages printed when early stopping is triggered are unconditional and should respect the verbose parameter.

The notebook changes are good, with a clear example of how to use the new early stopping feature and a fix to use val_loss consistently. After addressing the feedback, this will be a solid enhancement.

[gemini-code-assist]

This early stopping check is great, but it's missing from the training loop in the else block below (when sample_weight is None), which will prevent early stopping from working in that case. Please add a similar check there.

Also, this print statement is unconditional. It should only be executed if verbose > 0 to avoid unexpected output when verbose is set to 0.

Suggested change

	if self.stop_training:
	print("Training stopped with early stopping taking effect")
	break
	if self.stop_training:
	if verbose > 0:
	print("Training stopped with early stopping taking effect")
	break

[gemini-code-assist]

Similar to the check within the batch loop, this print statement is unconditional. It should be wrapped in an if verbose > 0: check to respect the verbosity level set for the fit method.

Suggested change

	if self.stop_training:
	print("Early Stopping taking effect")
	print("Training stopped with early stopping taking effect")
	break
	if self.stop_training:
	if verbose > 0:
	print("Training stopped with early stopping taking effect")
	break

[github-actions]

Coverage Report for Python 3.9

File	Stmts	Miss	Cover	Missing
choice_learn
__init__.py	2	0	100%
tf_ops.py	62	1	98%	283
choice_learn/basket_models
__init__.py	4	0	100%
alea_carta.py	148	22	85%	86–90, 92–96, 98–102, 106, 109, 131, 159, 308, 431–455
base_basket_model.py	235	27	89%	111–112, 123, 141, 185, 255, 377, 485, 585–587, 676, 762, 772, 822–830, 891–894, 934–935
basic_attention_model.py	89	4	96%	424, 427, 433, 440
self_attention_model.py	133	9	93%	71, 73, 75, 450–454, 651
shopper.py	184	9	95%	130, 159, 325, 345, 360, 363, 377, 489, 618
choice_learn/basket_models/data
__init__.py	2	0	100%
basket_dataset.py	190	30	84%	74–77, 295–297, 407, 540–576, 636, 658–661, 700–705, 790–801, 849
preprocessing.py	94	78	17%	43–45, 128–364
choice_learn/basket_models/datasets
__init__.py	3	0	100%
bakery.py	38	3	92%	47, 51, 61
synthetic_dataset.py	81	6	93%	62, 194–199, 247
choice_learn/basket_models/utils
__init__.py	0	0	100%
permutation.py	22	1	95%	37
choice_learn/data
__init__.py	3	0	100%
choice_dataset.py	649	33	95%	198, 250, 283, 421, 463–464, 589, 724, 738, 840, 842, 937, 957–961, 1140, 1159–1161, 1179–1181, 1209, 1214, 1223, 1240, 1281, 1293, 1307, 1346, 1361, 1366, 1395, 1408, 1443–1444
indexer.py	241	23	90%	20, 31, 45, 60–67, 202–204, 219–230, 265, 291, 582
storage.py	161	6	96%	22, 33, 51, 56, 61, 71
store.py	72	72	0%	3–275
choice_learn/datasets
__init__.py	4	0	100%
base.py	400	5	99%	42–43, 153–154, 714
expedia.py	102	83	19%	37–301
tafeng.py	49	0	100%
choice_learn/datasets/data
__init__.py	0	0	100%
choice_learn/models
__init__.py	14	2	86%	15–16
base_model.py	309	25	92%	144, 186, 283, 291, 297, 306, 346, 350–351, 356, 385, 389–390, 407, 438–439, 448–449, 550, 552, 568, 572, 574, 697–698
baseline_models.py	49	0	100%
conditional_logit.py	269	26	90%	49, 52, 54, 85, 88, 91–95, 98–102, 136, 206, 212–216, 351, 388, 445, 520–526, 651, 685, 822, 826
halo_mnl.py	124	2	98%	186, 374
latent_class_base_model.py	286	39	86%	55–61, 273–279, 288, 325–330, 497–500, 605, 624, 665–701, 715, 720, 751–752, 774–775, 869–870, 974
latent_class_mnl.py	62	6	90%	257–261, 296
learning_mnl.py	67	3	96%	157, 182, 188
nested_logit.py	291	12	96%	55, 77, 160, 269, 351, 484, 530, 600, 679, 848, 900, 904
reslogit.py	132	6	95%	285, 360, 369, 374, 382, 432
rumnet.py	236	3	99%	748–751, 982
simple_mnl.py	139	6	96%	167, 275, 347, 355, 357, 359
tastenet.py	94	3	97%	142, 180, 188
choice_learn/toolbox
__init__.py	0	0	100%
assortment_optimizer.py	27	6	78%	28–30, 93–95, 160–162
gurobi_opt.py	236	236	0%	3–675
or_tools_opt.py	230	11	95%	103, 107, 296–305, 315, 319, 607, 611
choice_learn/utils
metrics.py	85	43	49%	74, 126–130, 147–166, 176, 190–199, 211–232, 242
TOTAL	5618	841	85%

Tests	Skipped	Failures	Errors	Time
221	0 💤	0 ❌	0 🔥	7m 56s ⏱️

[github-actions]

Coverage Report for Python 3.10

File	Stmts	Miss	Cover	Missing
choice_learn
__init__.py	2	0	100%
tf_ops.py	62	1	98%	283
choice_learn/basket_models
__init__.py	4	0	100%
alea_carta.py	148	22	85%	86–90, 92–96, 98–102, 106, 109, 131, 159, 308, 431–455
base_basket_model.py	235	27	89%	111–112, 123, 141, 185, 255, 377, 485, 585–587, 676, 762, 772, 822–830, 891–894, 934–935
basic_attention_model.py	89	4	96%	424, 427, 433, 440
self_attention_model.py	133	9	93%	71, 73, 75, 450–454, 651
shopper.py	184	9	95%	130, 159, 325, 345, 360, 363, 377, 489, 618
choice_learn/basket_models/data
__init__.py	2	0	100%
basket_dataset.py	190	30	84%	74–77, 295–297, 407, 540–576, 636, 658–661, 700–705, 790–801, 849
preprocessing.py	94	78	17%	43–45, 128–364
choice_learn/basket_models/datasets
__init__.py	3	0	100%
bakery.py	38	3	92%	47, 51, 61
synthetic_dataset.py	81	6	93%	62, 194–199, 247
choice_learn/basket_models/utils
__init__.py	0	0	100%
permutation.py	22	1	95%	37
choice_learn/data
__init__.py	3	0	100%
choice_dataset.py	649	33	95%	198, 250, 283, 421, 463–464, 589, 724, 738, 840, 842, 937, 957–961, 1140, 1159–1161, 1179–1181, 1209, 1214, 1223, 1240, 1281, 1293, 1307, 1346, 1361, 1366, 1395, 1408, 1443–1444
indexer.py	241	23	90%	20, 31, 45, 60–67, 202–204, 219–230, 265, 291, 582
storage.py	161	6	96%	22, 33, 51, 56, 61, 71
store.py	72	72	0%	3–275
choice_learn/datasets
__init__.py	4	0	100%
base.py	400	5	99%	42–43, 153–154, 714
expedia.py	102	83	19%	37–301
tafeng.py	49	0	100%
choice_learn/datasets/data
__init__.py	0	0	100%
choice_learn/models
__init__.py	14	2	86%	15–16
base_model.py	309	25	92%	144, 186, 283, 291, 297, 306, 346, 350–351, 356, 385, 389–390, 407, 438–439, 448–449, 550, 552, 568, 572, 574, 697–698
baseline_models.py	49	0	100%
conditional_logit.py	269	26	90%	49, 52, 54, 85, 88, 91–95, 98–102, 136, 206, 212–216, 351, 388, 445, 520–526, 651, 685, 822, 826
halo_mnl.py	124	2	98%	186, 374
latent_class_base_model.py	286	39	86%	55–61, 273–279, 288, 325–330, 497–500, 605, 624, 665–701, 715, 720, 751–752, 774–775, 869–870, 974
latent_class_mnl.py	62	6	90%	257–261, 296
learning_mnl.py	67	3	96%	157, 182, 188
nested_logit.py	291	12	96%	55, 77, 160, 269, 351, 484, 530, 600, 679, 848, 900, 904
reslogit.py	132	6	95%	285, 360, 369, 374, 382, 432
rumnet.py	236	3	99%	748–751, 982
simple_mnl.py	139	6	96%	167, 275, 347, 355, 357, 359
tastenet.py	94	3	97%	142, 180, 188
choice_learn/toolbox
__init__.py	0	0	100%
assortment_optimizer.py	27	6	78%	28–30, 93–95, 160–162
gurobi_opt.py	238	238	0%	3–675
or_tools_opt.py	230	11	95%	103, 107, 296–305, 315, 319, 607, 611
choice_learn/utils
metrics.py	85	43	49%	74, 126–130, 147–166, 176, 190–199, 211–232, 242
TOTAL	5620	843	85%

Tests	Skipped	Failures	Errors	Time
221	0 💤	0 ❌	0 🔥	8m 36s ⏱️

[github-actions]

Coverage Report for Python 3.11

File	Stmts	Miss	Cover	Missing
choice_learn
__init__.py	2	0	100%
tf_ops.py	62	1	98%	283
choice_learn/basket_models
__init__.py	4	0	100%
alea_carta.py	148	22	85%	86–90, 92–96, 98–102, 106, 109, 131, 159, 308, 431–455
base_basket_model.py	235	27	89%	111–112, 123, 141, 185, 255, 377, 485, 585–587, 676, 762, 772, 822–830, 891–894, 934–935
basic_attention_model.py	89	4	96%	424, 427, 433, 440
self_attention_model.py	133	9	93%	71, 73, 75, 450–454, 651
shopper.py	184	9	95%	130, 159, 325, 345, 360, 363, 377, 489, 618
choice_learn/basket_models/data
__init__.py	2	0	100%
basket_dataset.py	190	30	84%	74–77, 295–297, 407, 540–576, 636, 658–661, 700–705, 790–801, 849
preprocessing.py	94	78	17%	43–45, 128–364
choice_learn/basket_models/datasets
__init__.py	3	0	100%
bakery.py	38	3	92%	47, 51, 61
synthetic_dataset.py	81	6	93%	62, 194–199, 247
choice_learn/basket_models/utils
__init__.py	0	0	100%
permutation.py	22	1	95%	37
choice_learn/data
__init__.py	3	0	100%
choice_dataset.py	649	33	95%	198, 250, 283, 421, 463–464, 589, 724, 738, 840, 842, 937, 957–961, 1140, 1159–1161, 1179–1181, 1209, 1214, 1223, 1240, 1281, 1293, 1307, 1346, 1361, 1366, 1395, 1408, 1443–1444
indexer.py	241	23	90%	20, 31, 45, 60–67, 202–204, 219–230, 265, 291, 582
storage.py	161	6	96%	22, 33, 51, 56, 61, 71
store.py	72	72	0%	3–275
choice_learn/datasets
__init__.py	4	0	100%
base.py	400	5	99%	42–43, 153–154, 714
expedia.py	102	83	19%	37–301
tafeng.py	49	0	100%
choice_learn/datasets/data
__init__.py	0	0	100%
choice_learn/models
__init__.py	14	2	86%	15–16
base_model.py	309	25	92%	144, 186, 283, 291, 297, 306, 346, 350–351, 356, 385, 389–390, 407, 438–439, 448–449, 550, 552, 568, 572, 574, 697–698
baseline_models.py	49	0	100%
conditional_logit.py	269	26	90%	49, 52, 54, 85, 88, 91–95, 98–102, 136, 206, 212–216, 351, 388, 445, 520–526, 651, 685, 822, 826
halo_mnl.py	124	2	98%	186, 374
latent_class_base_model.py	286	39	86%	55–61, 273–279, 288, 325–330, 497–500, 605, 624, 665–701, 715, 720, 751–752, 774–775, 869–870, 974
latent_class_mnl.py	62	6	90%	257–261, 296
learning_mnl.py	67	3	96%	157, 182, 188
nested_logit.py	291	12	96%	55, 77, 160, 269, 351, 484, 530, 600, 679, 848, 900, 904
reslogit.py	132	6	95%	285, 360, 369, 374, 382, 432
rumnet.py	236	3	99%	748–751, 982
simple_mnl.py	139	6	96%	167, 275, 347, 355, 357, 359
tastenet.py	94	3	97%	142, 180, 188
choice_learn/toolbox
__init__.py	0	0	100%
assortment_optimizer.py	27	6	78%	28–30, 93–95, 160–162
gurobi_opt.py	238	238	0%	3–675
or_tools_opt.py	230	11	95%	103, 107, 296–305, 315, 319, 607, 611
choice_learn/utils
metrics.py	85	43	49%	74, 126–130, 147–166, 176, 190–199, 211–232, 242
TOTAL	5620	843	85%

Tests	Skipped	Failures	Errors	Time
221	0 💤	0 ❌	0 🔥	9m 38s ⏱️

[github-actions]

Coverage Report for Python 3.12

File	Stmts	Miss	Cover	Missing
choice_learn
__init__.py	2	0	100%
tf_ops.py	62	1	98%	283
choice_learn/basket_models
__init__.py	4	0	100%
alea_carta.py	148	22	85%	86–90, 92–96, 98–102, 106, 109, 131, 159, 308, 431–455
base_basket_model.py	235	27	89%	111–112, 123, 141, 185, 255, 377, 485, 585–587, 676, 762, 772, 822–830, 891–894, 934–935
basic_attention_model.py	89	4	96%	424, 427, 433, 440
self_attention_model.py	133	9	93%	71, 73, 75, 450–454, 651
shopper.py	184	9	95%	130, 159, 325, 345, 360, 363, 377, 489, 618
choice_learn/basket_models/data
__init__.py	2	0	100%
basket_dataset.py	190	30	84%	74–77, 295–297, 407, 540–576, 636, 658–661, 700–705, 790–801, 849
preprocessing.py	94	78	17%	43–45, 128–364
choice_learn/basket_models/datasets
__init__.py	3	0	100%
bakery.py	38	3	92%	47, 53, 61
synthetic_dataset.py	81	6	93%	62, 194–199, 247
choice_learn/basket_models/utils
__init__.py	0	0	100%
permutation.py	22	1	95%	37
choice_learn/data
__init__.py	3	0	100%
choice_dataset.py	649	33	95%	198, 250, 283, 421, 463–464, 589, 724, 738, 840, 842, 937, 957–961, 1140, 1159–1161, 1179–1181, 1209, 1214, 1223, 1240, 1281, 1293, 1307, 1346, 1361, 1366, 1395, 1408, 1443–1444
indexer.py	241	23	90%	20, 31, 45, 60–67, 202–204, 219–230, 265, 291, 582
storage.py	161	6	96%	22, 33, 51, 56, 61, 71
store.py	72	72	0%	3–275
choice_learn/datasets
__init__.py	4	0	100%
base.py	400	5	99%	42–43, 153–154, 714
expedia.py	102	83	19%	37–301
tafeng.py	49	0	100%
choice_learn/datasets/data
__init__.py	0	0	100%
choice_learn/models
__init__.py	14	2	86%	15–16
base_model.py	309	25	92%	144, 186, 283, 291, 297, 306, 346, 350–351, 356, 385, 389–390, 407, 438–439, 448–449, 550, 552, 568, 572, 574, 697–698
baseline_models.py	49	0	100%
conditional_logit.py	269	26	90%	49, 52, 54, 85, 88, 91–95, 98–102, 136, 206, 212–216, 351, 388, 445, 520–526, 651, 685, 822, 826
halo_mnl.py	124	2	98%	186, 374
latent_class_base_model.py	286	39	86%	55–61, 273–279, 288, 325–330, 497–500, 605, 624, 665–701, 715, 720, 751–752, 774–775, 869–870, 974
latent_class_mnl.py	62	6	90%	257–261, 296
learning_mnl.py	67	3	96%	157, 182, 188
nested_logit.py	291	12	96%	55, 77, 160, 269, 351, 484, 530, 600, 679, 848, 900, 904
reslogit.py	132	6	95%	285, 360, 369, 374, 382, 432
rumnet.py	236	3	99%	748–751, 982
simple_mnl.py	139	6	96%	167, 275, 347, 355, 357, 359
tastenet.py	94	3	97%	142, 180, 188
choice_learn/toolbox
__init__.py	0	0	100%
assortment_optimizer.py	27	6	78%	28–30, 93–95, 160–162
gurobi_opt.py	238	238	0%	3–675
or_tools_opt.py	230	11	95%	103, 107, 296–305, 315, 319, 607, 611
choice_learn/utils
metrics.py	85	43	49%	74, 126–130, 147–166, 176, 190–199, 211–232, 242
TOTAL	5620	843	85%

Tests	Skipped	Failures	Errors	Time
221	0 💤	0 ❌	0 🔥	9m 19s ⏱️