Expected pretraining results (e.g., loss, next-sentence-prediction accuracy, etc.) #46

Would be great to get expected pretraining results (e.g., loss, next-sentence-prediction accuracy, etc.) and learning curves using the 64 * V100 training.

I found the fine-tuning task results on the blog but would be nice to compare the pretraining results as a "sanity check."

I don't have access to V100 GPUs for now so I'm using P40 GPUs to replicate/validate the pretraining pipeline.

Maxim Lukiyanov (@maxluk) Aashna Garg (@aashnamsft) any pretraining details that you guys can share? Even a screenshot of the AzureML metrics page would be helpful to compare and double-check.