You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Does tensor_parallel support multi-node tensor parallel training? #84
changed the title [-]Does tensor_parallel support multi-node training?[/-][+]Does tensor_parallel support multi-node tensor parallel training?[/+]on Jun 7, 2023
@BlackSamorez I have 2 servers with a total of 16 GPUs, so I would love to be able to use multi-nodes tensor-parallel to train a large language model, for example Bloom 176B. So I hope you can answer how to use multi-nodes tensor-parallel.
Thank you very much
ahaha everybody have same problem but I think there is no feature like this but we absolutely need it.
Recently I tried DeepSpeed which is developing from microsoft, maybe it has but Microsoft's code doesn't suppport Windows 😄
Activity
[-]Does tensor_parallel support multi-node training?[/-][+]Does tensor_parallel support multi-node tensor parallel training?[/+]zhangjunyi111 commentedon Aug 16, 2023
I want to konw too.
longday1102 commentedon Aug 27, 2023
@BlackSamorez Hope you can answer this question 😄😄
longday1102 commentedon Aug 27, 2023
@BlackSamorez I have 2 servers with a total of 16 GPUs, so I would love to be able to use multi-nodes tensor-parallel to train a large language model, for example Bloom 176B. So I hope you can answer how to use multi-nodes tensor-parallel.
Thank you very much
PieterZanders commentedon May 10, 2024
Is this solved? if so, how?
deema-A commentedon Aug 14, 2024
same question.
Tezcan98 commentedon Oct 8, 2024
ahaha everybody have same problem but I think there is no feature like this but we absolutely need it.
Recently I tried DeepSpeed which is developing from microsoft, maybe it has but Microsoft's code doesn't suppport Windows 😄