Skip to content

implement looking at multiple trials to enable stochastic models - #227

Merged
mschrimpf merged 7 commits into
masterfrom
trials
Aug 17, 2020
Merged

mschrimpf merged 7 commits into
masterfrom
trials

Conversation

@mschrimpf

@mschrimpf mschrimpf commented Jul 31, 2020 •

Copy link
Copy Markdown
Member
  • repeat stimulus_set with fixed number of repetitions
  • repeat stimulus_set with per-image repetitions
  • average model per-trial responses
  • update current benchmarks to use trials
  • ensure unit-tests pass
  • adjust model-tools for efficient trials
  • adjust model-tools for ImageNet assembly

@mschrimpf mschrimpf changed the title implement models looking at multiple trials to enable stochastic models implement looking at multiple trials to enable stochastic models Aug 3, 2020
@tiagogmarques

Copy link
Copy Markdown
Contributor

Majaj: ~50 repetitions
Ziemba: 20 repetitions
Rajalingham: ~2 repetitions
Cadena: “at least 2 repetitions” (if I remember in Brain-score you remove images that have NaNs so I think it leaves out only the images that had 4 repetitions but it may be worth revisiting
Kar: 44 repetitions

@mschrimpf

Copy link
Copy Markdown
Member Author

using 10 trials for ImageNet since that is what many model ensembles use

@mschrimpf
mschrimpf marked this pull request as ready for review August 4, 2020 19:33
@mschrimpf

Copy link
Copy Markdown
Member Author

@tiagogmarques approved

@mschrimpf
mschrimpf merged commit cddf89f into master Aug 17, 2020
@mschrimpf
mschrimpf deleted the trials branch May 15, 2022 22:25
Nishant-codex pushed a commit to Nishant-codex/vision that referenced this pull request Jul 2, 2026
…in-score#227)

* add integer repetition of trials

* implement average_trials

* set number of trials for all benchmarks

* mark number of trials test as private since it uses all benchmarks

* take max over trials for non-numeric; use assembly for ImageNet test

* ensure non-repetition coords have presentation dimension

e.g. squeezed time_bin shows up otherwise

* deal with stimulus_sets without a set meta attribute
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants