Failure while mpirun job depends on the order of the hosts

[karasevb]

Background information

What version of Open MPI are you using? (e.g., v1.10.3, v2.1.0, git branch name and hash, etc.)

v3.0.x
master

Describe how Open MPI was installed (e.g., from a source/distribution tarball, from a git clone, from an operating system distribution package, etc.)

git clone

./configure --prefix=`pwd`/install --enable-orterun-prefix-by-default --with-slurm --with-pmi --with-ucx

Please describe the system on which you are running

• Operating system/version:
  RedHat 7.2
• Computer hardware:
  Intel dual socket Broadwell
• Network type:
  IB

---

Details of the problem

Running on nodes node1,node2 works well, but if change the order of the nodes to node2,node1 this will result to failure:

ssh node1
mpirun --bind-to core --map-by node -H node2,node1 -np 2  $HPCX_MPI_DIR/tests/osu-micro-benchmarks-5.3.2/osu_allreduce
--------------------------------------------------------------------------
[node2:13941] Error: pml_yalla.c:95 - recv_ep_address() Failed to receive EP address
--------------------------------------------------------------------------
It looks like MPI_INIT failed for some reason; your parallel process is
likely to abort.  There are many reasons that a parallel process can
fail during MPI_INIT; some of which are due to configuration or environment
problems.  This failure appears to be an internal failure; here's some
additional information (which may only be relevant to an Open MPI
developer):

  PML add procs failed
  --> Returned "Not found" (-13) instead of "Success" (0)

[karasevb]

This issue was introduced with commit fe9b584

[jsquyres]

@karasevb FYI: I filed a dup of this in #4726 (and closed it; leaving this one open). Just mentioning it here as a cross reference -- when this issue is fixed, we should check to make sure the case sited in #4726 works as well.

[rhc54]

FWIW: I found the problem. The daemon on node2 incorrectly assigns rank=0 to node1 and rank=1 to itself, while the daemon on node1 (correctly) assigns rank=0 to node2 and rank=1 to itself. Thus, the daemons start two rank=1 procs, and no rank=0 as they each think the other one launched it.

[jsquyres]

@rhc54 Cool -- thanks! Would this also explain what was happening in #4726?

@karasevb Can you fix this, perchance?

[rhc54]

Yeah. The race condition is due to the shortness of "hostname" execution. If it finishes fast enough on the first node, then the second one receives notification that the other proc died and correctly computes its local rank. You can reproduce it 100% of the time by using a longer-running executable.

[jsquyres]

I did some testing today: I ran this scenario:

master$ ssh hostA
hostA$ mpirun --hostA,hostB sleep 1
hostA$ mpirun --hostB,hostA sleep 1

Per @rhc's comment #4516 (comment), running hostname is a test program that is short enough to make the problem be a race condition. But invoking a longer test program should make the problem occur 100% of the time. So I ran sleep 1 for my tests today, which should be long enough to make it happen 100% of the time.

Results:

• On master: problem occurs 100% of the time, as predicted.
• On v3.0.0: cannot reproduce the problem after 100 runs.
• On v3.0.x head: cannot reproduce the problem after 100 runs.
• On v3.x head: cannot reproduce the problem after 100 runs.

I also checked: my original testing (on #4726) was on master, not any of the v3.x.y branches.

So AFAICT, this problem exists on master but does not exist (or no longer exists?) on any of the 3.x.y release branches.

@jladd-mlnx @karasevb Just because I can't reproduce it doesn't mean that the problem isn't still there on the v3.x.y branches (as indicated by @karasevb's initial report). Can you try to reproduce on the v3.x.y branches?

If no one can reproduce on the v3.x.y branches, then we just change the labels/milestone on this issue.

[gpaulsen]

I tested with an older open-mpi in v3.0.x, and I couldn't get it to reproduce at all on ppc64le.
I also saw in master the problem occur 100% of the time on ppc64le.

Based on Jeff's testing, I'm removing blocker and targets other than master.