libnbc nonuniform type failures in ibcast, iallgather(v)

[jjhursey]

When testing coll/libnbc with nonuniform data types three of the algorithms failed. I'll post a link to the test case in the comments. The unit test only works with -np 5 some of the tests pass fine with -np 4.

• ibcast Falls into an infinite error loop in libnbc
• iallgather and iallgatherv produce wrong answers. Might be the same underlying problem.

ibcast Failure

Note: This test will enter an infinite loop displaying MPI Error in MPI_Testall() (18) until PR #2245 is resolved.

shell$ mpirun -np 5 --mca coll ^hcoll  ./test-nbc-dt 0 
 0 /  5) Running MPI_Ibcast...
 1 /  5) Running MPI_Ibcast...
 2 /  5) Running MPI_Ibcast...
 3 /  5) Running MPI_Ibcast...
 4 /  5) Running MPI_Ibcast...
MPI Error in MPI_Testall() (18)
MPI Error in MPI_Testall() (18)
...

iallgather Failure

shell$ mpirun -np 5 --mca coll ^hcoll  ./test-nbc-dt 1
 0 /  5) Running MPI_Iallgather...
 1 /  5) Running MPI_Iallgather...
 2 /  5) Running MPI_Iallgather...
 3 /  5) Running MPI_Iallgather...
 4 /  5) Running MPI_Iallgather...
 3 /  5) buf[1] : actual 2 [2], expected 1 [1]. from 0
 3 /  5) Failed in chkbuf. err = -1
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 3 in communicator MPI_COMM_WORLD
with errorcode -2.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
 2 /  5) buf[1] : actual 2 [2], expected 1 [1]. from 0
 2 /  5) Failed in chkbuf. err = -6
 1 /  5) buf[2] : actual 2 [2], expected 1 [1]. from 0
 1 /  5) Failed in chkbuf. err = -2
 4 /  5) buf[1] : actual 2 [2], expected 1 [1]. from 0
 4 /  5) Failed in chkbuf. err = -6

iallgatherv Failure

shell$ mpirun -np 5 --mca coll ^hcoll  ./test-nbc-dt 2
 0 /  5) Running MPI_Iallgatherv...
 1 /  5) Running MPI_Iallgatherv...
 2 /  5) Running MPI_Iallgatherv...
 3 /  5) Running MPI_Iallgatherv...
 4 /  5) Running MPI_Iallgatherv...
 0 /  5) buf[1] : actual 3 [3], expected 2 [2]. from 1
 0 /  5) Failed in chkbuf. err = -6
 3 /  5) buf[1] : actual 3 [3], expected 2 [2]. from 1
 3 /  5) Failed in chkbuf. err = -6
 4 /  5) buf[1] : actual 3 [3], expected 2 [2]. from 1
 4 /  5) Failed in chkbuf. err = -6
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 3 in communicator MPI_COMM_WORLD
with errorcode -2.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
 2 /  5) buf[1] : actual 3 [3], expected 2 [2]. from 1
 2 /  5) Failed in chkbuf. err = -6

[jjhursey]

Here is the test case for this issue:
https://gist.github.com/jjhursey/dfbb38c7c82dcde7d0e05e62852fce3c

Sorry it's not more general, but should help reproduce the problem.

[ggouaillardet]

@jjhursey i fixed MPI_Iallgather[v] in #2245

the issue with MPI_Ibcast is the very same and unresolved issue we have in coll/tuned.
we should not try to split a message into chunks since we cannot be sure the remote peer is able to split the message into chunks of the same size.
for example, if you need to MPI_Send 1M MPI_INT, it is trivial to MPI_Send 1k MPI_INT 1024 times (and this is what we do in ibcast and coll/tuned
but if the remote peer MPI_Recv one vector of 1M MPI_INT, there is currently no way to split that into 1024 MPI_Recv of partial vector.
we currently assume we do not run into this case and the remote peer will be just fine, and once in a while, we are reminded the assumption is wrong.
fwiw, i started to work on #1789 as a proof of concept, but had not much time to finalize it

[jjhursey]

@ggouaillardet I can confirm that your commit fixed the problem with the MPI_Iallgather and MPI_Iallgatherv collectives. Thanks!

Adding a reference to Issue #1763 - which describes the data corruption problem with MPI_Bcast. Since those two are related problems, a fix for one should be ported to the other.

As a workaround I might look into adding an MCA parameter to skip the algorithm selection based upon message size for bcast.

[ggouaillardet]

@jjhursey IIRC, coll/tuned has a (per collective ?) segment_size MCA param, and we recommend it is set to 0 when running into this kind of pattern. You might want to use the same naming scheme for libnbc.
An other workaround is to blacklist coll/tuned, but that cannot be mimicked for non-blocking collectives.