MPI changes for compatibility with MVAPICH2

[ponyisi@utexas.edu]

Hi - I have been trying to get Sherpa 1.4.3 running in MPI mode on our local system (uses MVAPICH2). I found the following changes necessary:

In Sherpa.C, Sherpa::InitializeTheRun:

• accessing the hostname by rpa->gen.Variable("HOSTNAME") needed to be replaced by MPI_Get_processor_name (I see this has already landed in 2.0.0);
• the code doesn't handle the edge case where MPI_COMBINE_PROCS=1 properly (basically because the code assumes it >= 2), fixed with adding

"if (cprocs[host] == ppn) cprocs[host] = 0;" after line 119, and moving "if (cprocs[hosts[tag]]==ppn) cprocs[hosts[tag]]=0;" from line 143 _outside_ the else { }. (Would also be fixed with modular arithmetic.)

In Exception_Handler.C, Exception_Handler::MPISuspend:

• I found it necessary to replace the "MPI_Irsend" by "MPI_Send", or else I got errors about how there was no receive available for the ready send. The blocking send works fine.

Best, Peter Onyisi

[Stefan Hoeche]

Hi Peter,

thank you for the suggestions. We will improve MPI support in version 2.0.0. However, the switch MPI_COMBINE_PROCS will be removed, as it is useful only on small, very heterogenous systems. Running more than one thread per MPI node will still be possible by specifying PG_THREADS=<#threads>, but allocation will not be dynamic. At the same time, we will cease to support automatic load balancing (this may be re-implemented at a later stage, if necessary).

Best

Stefan