best_practice_torch_distribute_barrier

This repository is made to providie a best practice for torch.distribute barrier among multiple nodes

cite : https://pytorch.org/docs/master/distributed.html#torch.distributed.new_group

https://zhuanlan.zhihu.com/p/76638962

https://pytorch.org/tutorials/intermediate/ddp_tutorial.html

https://pytorch.org/docs/stable/distributed.html