How can we update the network weight when the parameters are synchronized?

[jacklonghui]

Hi，
When the parameters are synchronized, each node sends the weights in its own parameter server to other nodes and receives the weights sent from all other nodes. now, each node has the same cache, storing the weight of the number of nodes.
We know that CaffeOnSpark will only retain a weight, as an update of the weight value, which will be used in the forward propagation of the next iteration. Then, how do you choose a weight update from the weight of the number of nodes as the weight of the next iteration?
Of course, perhaps I understand the weight of each node's sending and receiving is wrong. My view comes from the source code CaffeOnSpark/caffe-distri/include/util/*
How does CaffeOnSpark update the weight? Please help me, thank you!

[junshi15]

#262

[jacklonghui]

@junshi15
Thank you for your answer. Now I know the way to synchronize the parameters.
Can you tell me the location of the detailed source code for parameter synchronization? I want to combine source code to understand parameter synchronization.
I'm interested in the synchronization of CaffeOnSpark.
About the parameters synchronization, I read the source code of CaffeOnSpark/caffe-distri/include/util/* and CaffeOnSpark/caffe-distri/src/main/cpp/util/*, understand the connection between nodes and communication

[junshi15]

https://github.com/yahoo/CaffeOnSpark/blob/master/caffe-distri/src/main/cpp/util/rdma_sync.cpp
https://github.com/yahoo/CaffeOnSpark/blob/master/caffe-distri/src/main/cpp/util/socket_sync.cpp
https://github.com/yahoo/CaffeOnSpark/blob/master/caffe-distri/src/main/cpp/util/socket_sync_cpu.cpp

look for on_gradients_ready() and on_start()

[jacklonghui]

@junshi15 Ok, thanks! May I send an e-mail to you in Chinese?
Because I have some problems that have been bothering me and cannot express myself accurately in English.
In addition, from this part of the source code, I drew a graph of the parameters synchronization between the nodes.
I hope you can help me correct it. If possible, please send your email address to my email address at [email protected].