Repository navigation
run_squad questions #3
Description
Thanks a lot for the port! I have some minor questions, for the run_squad file, I see two options for accumulating gradients, accumulate_gradients and gradient_accumulation_steps but it seems to me that it can be combined into one. The other one is for the global_step variable, seems we are only counting but not using this variable in gradient accumulating. Thanks again!
Reacted by Mohd Aasim Ansari
Metadata
Metadata
Assignees
Labels
No labels
Activity
ZhaoyueCheng commentedon Nov 5, 2018
It also seems to me that the SQuAD 1.1 can not reproduce the google tensorflow version performance.
abeljim commentedon Nov 5, 2018
What batch size are you running?
ZhaoyueCheng commentedon Nov 6, 2018
I'm running on 4 GPU with a batch size of 48, the result is {"exact_match": 21.551561021759696, "f1": 41.785968963154055}
abeljim commentedon Nov 6, 2018
Just ran on 1 GPU batch size of 10, the result is {"exact_match": 21.778618732261116, "f1": 41.83593185416649}
Actually it might be with the eval code Ill look into it
ZhaoyueCheng commentedon Nov 6, 2018
Sure, Thanks, I'm checking for the reason too, will report if find anything.
abeljim commentedon Nov 6, 2018
The predictions file is only outputting one word. Need to find out if the bug is in the model itself or write predictions function in run_squad.py. The correct answer always seems to be in the nbest_predictions, but its never selected.
[-]Minor run_squad questions[/-][+]run_squad questions[/+]ethanjperez commentedon Nov 6, 2018
What performance does Hugging Face get on SQuAD using this reimplementation?
thomwolf commentedon Nov 6, 2018
Hi all,
We were not able to try SQuAD on a multi-GPU with the correct batch_size until recently so we relied on the standard deviations computed in the notebooks to compare the predicted hidden states and losses for the SQuAD script. I was able to try on a multi-GPU today and there is indeed a strong difference.
We got about the same results that you get: F1 of 41.8 and exact match of 21.7.
I am investigating that right now, my personal guess is that this may be related to things outside the model it-self like the optimizer or the post-processing in SQuAD as these were not compared between the TF and PT models.
I will keep you guys updated in this issue and I add a mention in the readme that the SQuAD example doesn't work yet.
If you have some insights, feel free to participate in the discussion.
ethanjperez commentedon Nov 6, 2018
If you're comparing activations, it may be worth comparing gradients as well to see if you receive similarly low gradients standard deviations for identical batches. You might see that the gradient is not comparable from the last layer itself (due to e.g. difference in how PyTorch may handle weight decay / optimization differently); you may also see that gradients only become not comparable only after a particular point in backpropagation, and that would show perhaps that the backward pass for a particular function differs between PyTorch and Tensorflow
thomwolf commentedon Nov 7, 2018
Ok guys thanks for waiting, we've nailed down the culprit which was in fact a bug in the pre-processing logic (more exactly this dumb typo https://github.com/huggingface/pytorch-pretrained-BERT/blob/master/run_squad.py#L865).
I took the occasion to clean up a few things I noticed while walking through the code:
tf. truncated_normal_initializer(stddev=0.02)was translated inweight.data.normal_(0.02)instead ofweight.data.normal_(mean=0.0, std=0.02)which likely affected the performance ofrun_classifer.pyalso.torch.no_grad()and thus sub-optimal in terms of speed/memory.These fixes are pushed on the
developbranch right now.All in all I think we are pretty good now and none of these issues affected the core PyTorch model (the BERT Transformer it-self) so if you only used
extract_features.pyyou were good from the beginning. Andrun_classifer.pywas ok apart from the sub-optimal additional weights initialization.I will merge the develop branch as soon as we got the final results confirmed (currently it's been training for 20 minutes (0.3 epoch) on 4GPU with a batch size of 56 and we are already above 85 on F1 on SQuAD and 77 in exact match so I'm rather confident and I think you guys can play with it too now).
I am also cleaning up the code base to prepare for a first release that we will put on pip for easier access.
ethanjperez commentedon Nov 8, 2018
@thomwolf This is awesome - thank you! Do you know what the final SQuAD results were from the training run you started?
86 remaining items
Revert "Fix DeepSpeed mixed precision precedence over Accelerate defa…
Revert "Fix DeepSpeed mixed precision precedence over Accelerate defa…
v4.57.0 Branch (#41310)
Merge pull request huggingface#3 from kashif/alst-doc
Clean up huggingface#3
Revert cu_seqlens synthesis in _flash_attention_forward
Revert cu_seqlens synthesis in _flash_attention_forward
modeling_flash_attention_utils: varlen fallback when flash_attn_func …
GGUF cleanup pass: drop modeling_utils side-path + fix MoE config att…
Enable kernels-community/metal-flash-sdpa on MPS (#45974)
Enable kernels-community/metal-flash-sdpa on MPS (huggingface#45974)
Enable kernels-community/metal-flash-sdpa on MPS (huggingface#45974)
Enable kernels-community/metal-flash-sdpa on MPS (huggingface#45974)
Enable kernels-community/metal-flash-sdpa on MPS (huggingface#45974)
Fix more stuff (huggingface#48979)