Excessive time spent on MPI Bcast in nnp-train

[mikejwaters]

I am running n2p2 on NERSC's newest, machine Perlmutter (https://docs.nersc.gov/performance/readiness/) using their PrgEnv-gnu compiler wrappers (https://docs.nersc.gov/systems/perlmutter/software/#compilers). After seeing terrible performance, the NERSC staff and I profiled the performance and found 98% of the time was spent on MPI with 96.4% of time spent on MPI_Bcast.

While the NERSC staff continues to work their vendors software team, is there anything I can try to mitigate this?

Edit: I'm on the master branch.

[singraber]

Hello!

What kind of run did you execute? Was it nnp-train or a LAMMPS/n2p2 run? What kind of nodes and how many cores did you use? Please provide as much information as you have otherwise I can only speculate...

Best,
Andreas