MPI_Waitall crashes with mixed ISend IRecv requests

[chhu]

Background information

After a recent update of ompi caused our simulation and benchmark crashing unexpectedly on MPI_Waitall.

[athos:00000] *** An error occurred in MPI_Waitall
[athos:00000] *** reported by process [3991207937,1]
[athos:00000] *** on communicator MPI_COMM_WORLD
[athos:00000] *** MPI_ERR_REQUEST: invalid request
[athos:00000] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
[athos:00000] *** and MPI will try to terminate your MPI job as well)

This was working before and after experimenting I found a fix by issuing two separate MPI_Waitall on the ISend reqs and IRecv reqs. So my suspicion is the mix.
The little benchmark code is attached.

WARNING: This is one of the bugs that I cannot safely reproduce. In 3 out of 10 times it actually works... :(

What version of Open MPI are you using? (e.g., v4.1.6, v5.0.1, git branch name and hash, etc.)

mpirun --version
mpirun (Open MPI) 5.1.0a1
(recent pull on main)

Describe how Open MPI was installed (e.g., from a source/distribution tarball, from a git clone, from an operating system distribution package, etc.)

mkdir ucx_build
git clone https://github.com/openucx/ucx.git
cd ucx
./autogen.sh
contrib/configure-release-mt --enable-optimizations --with-mlx5 --with-march --with-cuda=$CUDA_PATH --with-gdrcopy=$P/gdrcopy_build --with-mlx5-dv --prefix=$P/ucx_build
make V=1 -j
make install
cd ..
mkdir ompi_build
git clone --depth 1 https://github.com/open-mpi/ompi.git --recursive
cd ompi
./autogen.pl
./configure --prefix=$P/ompi_build --with-cuda=$CUDA_PATH --with-ucx=$P/ucx_build
make -j
make install

If you are building/installing from a git clone, please copy-n-paste the output from git submodule status.

08e41ed 3rd-party/openpmix (v1.1.3-4067-g08e41ed5)
30cadc6746ebddd69ea42ca78b964398f782e4e3 3rd-party/prrte (psrvr-v2.0.0rc1-4839-g30cadc6746)
dfff67569fb72dbf8d73a1dcf74d091dad93f71b config/oac (dfff675)

Please describe the system on which you are running

• Operating system/version: RH8
• Computer hardware: Epyc 1stgen w/ CUDA
• Network type: MLX5

---

Details of the problem

mpicxx  -O2 -std=c++17 -march=native -w -DUSE_MPI gflops.cpp -o gflops
mpirun -n 4 ./gflops

gflops.cpp:

#include <stdio.h>
#include <iostream>
#include <string>
#include <algorithm>
#include <numeric>
#include <vector>
#include <valarray>
#include <chrono>
//#include <cblas.h>
#ifdef USE_MPI
#include <mpi.h>
#endif

using namespace std;
using namespace chrono;


double dot(const vector<double> &a, const vector<double> &b) {

	//return std::inner_product(a.begin(), a.end(), b.begin(), 0); // Comparison to STLs own dot product. Very slow.
	//return cblas_ddot (a.size(), &(a[0]), 1, &(b[0]), 1);		 // Comparison to cblas / MKL / BLIS. No speed gain.
	double result = 0;
	for (size_t i = 0; i < a.size(); i += 8)
		result += a[i + 0] * b[i + 0] + a[i + 1] * b[i + 1]	// Vectorizing hint, should produce AVX512 machine code where possible
	        	+ a[i + 2] * b[i + 2] + a[i + 3] * b[i + 3]
			+ a[i + 4] * b[i + 4] + a[i + 5] * b[i + 5]
			+ a[i + 6] * b[i + 6] + a[i + 7] * b[i + 7];
	return result;
}

int main(int argc, char **argv) {

	size_t mem_size = 0;
	unsigned loops = 0;
	int rank = 0, cpu_count = 1;
#ifdef USE_MPI
	MPI_Init( &argc, &argv );
	MPI_Comm communicator = MPI_COMM_WORLD;

	MPI_Comm_rank(communicator, &rank);
	MPI_Comm_size(communicator, &cpu_count);
	if (rank == 0)
		printf("MPI Enabled, node %d of %d\nOutput is [min of single rank, max of single rank, TOTAL]\nThis benchmark does not account for MPI overhead. ", rank, cpu_count);
#endif
	if (rank == 0)
		printf("A large dot product is performed that quickly evades CPU cache.\n");

	if (argc != 3) {
		if (rank == 0)
			cout << "Use: ./gflops <MByte to test per rank> <N loops>\nUsing default: 200MB / rank, 30 loops.\n";
		mem_size = 200;
		loops = 30;

	} else {
		mem_size = (size_t) stoi(string(argv[1]), nullptr);
		loops = stoi(string(argv[2]), nullptr);
	}
	mem_size *=  1024 * 1024;

	size_t elements = mem_size / sizeof(double) / 2;
	vector<double> best_gflop;
	vector<double> a(elements, 0.);
	vector<double> b(elements, 0.);

	for (size_t i = 0; i < elements; i++) {
		a[i] = double(elements) / (i + 1.);
		b[i] = 1. - a[i];
	}

	if (rank == 0)
		cout << "Dot product with 2 " << elements << " double arrays per rank\n";

#ifdef USE_MPI
	double single_ref_gflop = 1;
	if (rank == 0) {
		auto t1 = high_resolution_clock::now();
		volatile double result = 0;
		result += dot(a,b);
		result += dot(a,b);
		result += dot(a,b);
		result += dot(a,b);
		auto t2 = high_resolution_clock::now();
	
		auto duration_i = duration_cast<microseconds>(t2-t1).count();
		double duration = double(duration_i) / 1e6 / 4.;  // 4 times loop

		single_ref_gflop = (1e-9) * ((elements) / (double)duration);
		printf("Single reference for speedup: %5.3g GFLOPs\n", single_ref_gflop);
	}
	MPI_Barrier(communicator);
#endif

	int n_loops = loops;
	while (loops--) {

		volatile double result = 0; // Avoid optimization

		auto t1 = high_resolution_clock::now();

		result += dot(a, b);
		result += dot(a, b);
		result += dot(a, b);
		result += dot(a, b);

		auto t2 = high_resolution_clock::now();

		auto duration_i = duration_cast<microseconds>(t2 - t1).count();
		double duration = double(duration_i) / 1e6 / 4.;  // 4 times loop

		double gflops = (1e-9) * ((elements) / (double) duration);

		double mem_speed =
				double(elements) * 2. * double(sizeof(double))
						/ double(duration) / 1024. / 1024. / 1024.; // GB/s
#ifdef USE_MPI
		// Get min/max/sum from all ranks
		double send_data[2] = {gflops, mem_speed};
		double receive_data_sum[2];
		double receive_data_min[2];
		double receive_data_max[2];
		MPI_Allreduce(send_data, receive_data_sum, 2, MPI_DOUBLE, MPI_SUM, communicator);
		MPI_Allreduce(send_data, receive_data_min, 2, MPI_DOUBLE, MPI_MIN, communicator);
		MPI_Allreduce(send_data, receive_data_max, 2, MPI_DOUBLE, MPI_MAX, communicator);
		best_gflop.push_back(receive_data_sum[0]);
		if (rank == 0)
			printf("Run %u: [%5.3g, %5.3g, %5.3g] GFLOP/s (FMA is 1 FLOP), [%5.3g, %5.3g, %5.3g] GByte/s. Speedup: %.4g\n", n_loops - loops, receive_data_min[0],  receive_data_max[0], receive_data_sum[0],  receive_data_min[1], receive_data_max[1],  receive_data_sum[1], receive_data_sum[0] / single_ref_gflop);
#else
		best_gflop.push_back(gflops);
		if (rank == 0)
			printf("Run %u: %5.3g GFLOP/s (FMA is 1 FLOP), %5.3g GByte/s.\n", n_loops - loops, gflops, mem_speed);
#endif
	}
	if (rank == 0) {
		printf("Peak performance: %g GFLOP/s.\n",
				*max_element(best_gflop.begin(), best_gflop.end()));
		printf("Average performance: %g GFLOP/s.\n",
				accumulate(best_gflop.begin(), best_gflop.end(), 0.) / best_gflop.size());
	}
#ifdef USE_MPI
	// source: https://www.mcs.anl.gov/research/projects/mpi/tutorial/mpiexmpl/src3/pingpong/C/nbhead/main.html
	double *sbuf, *rbuf;
	int n;
	double t1, t2;
	valarray<double> tmin(0.0, cpu_count);
	int i, j, k, nloop;
	MPI_Status status, statuses[2];
	MPI_Request r[2] = {MPI_REQUEST_NULL};

	if (rank == 0)
		printf("\nMPI-Latency and Bandwidth Test\nKind\t\t\t\tbytes\ttime (sec)\tRate (MB/sec)\n");

	for (n = 1; n < 1100000*16; n *= 2) {
		if (n == 0)
			nloop = 1000;
		else
			nloop = 1000 / n;
		if (nloop < 1)
			nloop = 1;

		sbuf = (double*) malloc(n * sizeof(double));
		rbuf = (double*) malloc(n * sizeof(double));

		if (!sbuf || !rbuf) {
			fprintf( stderr,"Could not allocate send/recv buffers of size %d\n", n);
			MPI_Abort(MPI_COMM_WORLD, 1);
		}
		for (k = 0; k < 10; k++) {
			if (rank == 0) {
				for (int target = 1; target < cpu_count; target++) {
					tmin[target] = 1e8;
					/* Make sure both processes are ready */
					MPI_Sendrecv(MPI_BOTTOM, 0, MPI_INT, target, 14, MPI_BOTTOM, 0, MPI_INT, target, 14, MPI_COMM_WORLD, &status);
					t1 = MPI_Wtime();

					for (j = 0; j < nloop; j++) {
						MPI_Isend(sbuf, n, MPI_DOUBLE, target, k, MPI_COMM_WORLD, &r[0]);
						MPI_Irecv(rbuf, n, MPI_DOUBLE, target, k, MPI_COMM_WORLD, &r[1]);
						MPI_Waitall(2, r, MPI_STATUSES_IGNORE);  // crashes !
						//MPI_Wait(&r[0], MPI_STATUS_IGNORE);  // works
						//MPI_Wait(&r[1], MPI_STATUS_IGNORE);
					}
					t2 = (MPI_Wtime() - t1) / nloop;
					if (t2 < tmin[target])
						tmin[target] = t2;
				}
			} else {
				/* Make sure both processes are ready */
				MPI_Sendrecv(MPI_BOTTOM, 0, MPI_INT, 0, 14, MPI_BOTTOM, 0, MPI_INT, 0, 14, MPI_COMM_WORLD, &status);
				for (j = 0; j < nloop; j++) {
					MPI_Irecv(rbuf, n, MPI_DOUBLE, 0, k, MPI_COMM_WORLD, &r[0]);
					MPI_Isend(sbuf, n, MPI_DOUBLE, 0, k, MPI_COMM_WORLD, &r[1]);
					//MPI_Wait(&r[0], MPI_STATUS_IGNORE);  // works!
					//MPI_Wait(&r[1], MPI_STATUS_IGNORE);  
					MPI_Waitall(2, r, MPI_STATUSES_IGNORE);  // crashes!
				}
			}
			
		}

		/* rate is MB/sec for exchange, not an estimate of the
		 component for each isend/irecv */
		if (rank == 0) {
			double rate;
			double tmin_ = tmin.max();
			rate = tmin_ == 0 ? 0 : 2 * n * sizeof(double) * 1.0e-6 / tmin_;
			printf("Worst head-to-head Isend/Irecv\t%d\t%g\t%f\n", n, tmin_, rate);
		}
		free(sbuf);
		free(rbuf);
	}
	MPI_Finalize();
#endif
}

[ggouaillardet]

Thanks for the report!
are you running on a single node? if not, which interconnect are you using?

[chhu]

Glad If I can help. Interconnect is mlx5 (Infiniband 40G), but the problem is independent, happens on shm as well.

[ggouaillardet]

i tried to reproduce this on a non CUDA system, but failed.

can you please apply the following patch and share the output when it fails?

Also, does the test (occasionally) fails if you mpirun -np 4 --mca pml ob1 ./gflops?

diff --git a/ompi/mpi/c/waitall.c b/ompi/mpi/c/waitall.c
index ce2353dfc9..b95e6a0029 100644
--- a/ompi/mpi/c/waitall.c
+++ b/ompi/mpi/c/waitall.c
@@ -59,6 +59,7 @@ int MPI_Waitall(int count, MPI_Request requests[], MPI_Status statuses[])
         int rc = MPI_SUCCESS;
         OMPI_ERR_INIT_FINALIZE(FUNC_NAME);
         if( (NULL == requests) && (0 != count) ) {
+            fprintf(stderr, "MPI_Waitall(%d, NULL, ...)\n", count);
             rc = MPI_ERR_REQUEST;
         } else {
             if(!ompi_request_check_same_instance(requests, count) ) {
diff --git a/ompi/request/request.c b/ompi/request/request.c
index 843d9ae95f..154677383a 100644
--- a/ompi/request/request.c
+++ b/ompi/request/request.c
@@ -267,8 +267,20 @@ bool ompi_request_check_same_instance(ompi_request_t** requests,
             base_instance = req->req_mpi_object.comm->instance;
             continue;
         }
-        if(base_instance != req->req_mpi_object.comm->instance)
+        if(base_instance != req->req_mpi_object.comm->instance) {
+            fprintf(stderr, "requests from different instances: %p != %p\n", base_instance, req->req_mpi_object.comm->instance);
+            fprintf(stderr, "MPI_COMM_WORLD instance is %p\n", ompi_mpi_comm_world.comm.instance);
+            for (int i=0; i<count; i++) {
+                if (OMPI_REQUEST_NULL == req->req_type) {
+                    fprintf(stderr, "request %d/%d is OMPI_REQUEST_NULL\n", i, count);
+                } else if (OMPI_REQUEST_PML != req->req_type) {
+                    fprintf(stderr, "request %d/%d is not OMPI_REQUEST_PML\n", i, count);
+                } else {
+                    fprintf(stderr, "request %d/%d instance is %p\n", i, count, req, req->req_mpi_object.comm->instance);
+                }
+            }
             return false;
+        }
     }
     return true;
 }

[chhu]

First I have to add a correction. Waitall also fails if I split the requests if I use a bigger setup with more nodes (InfiniBand + Shm).

With the --mca pml ob1 everything works! Although I don't know what it does... I will add the patch and see what happens.

[chhu]

Here is the output:

MPI Enabled, node 0 of 315
Output is [min of single rank, max of single rank, TOTAL]
This benchmark does not account for MPI overhead. A large dot product is performed that quickly evades CPU cache.
Dot product with 2 65536000 double arrays per rank
Single reference for speedup:  1.52 GFLOPs
Run 1: [0.271,     1,   179] GFLOP/s (FMA is 1 FLOP), [ 4.04,  14.9, 2.66e+03] GByte/s. Speedup: 117.4
Run 2: [0.271,  1.01,   179] GFLOP/s (FMA is 1 FLOP), [ 4.04,    15, 2.66e+03] GByte/s. Speedup: 117.4
Run 3: [0.271,     1,   178] GFLOP/s (FMA is 1 FLOP), [ 4.05,    15, 2.66e+03] GByte/s. Speedup: 117.2
Run 4: [0.272,     1,   179] GFLOP/s (FMA is 1 FLOP), [ 4.05,    15, 2.67e+03] GByte/s. Speedup: 117.5
Run 5: [0.272,  1.01,   178] GFLOP/s (FMA is 1 FLOP), [ 4.05,    15, 2.66e+03] GByte/s. Speedup: 117.2
Run 6: [0.272,     1,   179] GFLOP/s (FMA is 1 FLOP), [ 4.05,    15, 2.67e+03] GByte/s. Speedup: 117.5
Run 7: [0.272,  1.01,   179] GFLOP/s (FMA is 1 FLOP), [ 4.05,    15, 2.66e+03] GByte/s. Speedup: 117.3
Run 8: [0.272,  1.01,   179] GFLOP/s (FMA is 1 FLOP), [ 4.05,    15, 2.66e+03] GByte/s. Speedup: 117.2
Run 9: [0.272,     1,   179] GFLOP/s (FMA is 1 FLOP), [ 4.05,    15, 2.66e+03] GByte/s. Speedup: 117.3
Run 10: [0.272,  1.01,   179] GFLOP/s (FMA is 1 FLOP), [ 4.05,    15, 2.66e+03] GByte/s. Speedup: 117.4
Peak performance: 178.994 GFLOP/s.
Average performance: 178.699 GFLOP/s.

MPI-Latency and Bandwidth Test
Kind				bytes	time (sec)	Rate (MB/sec)
requests from different instances: 0x7ff9a802cd10 != (nil)
MPI_COMM_WORLD instance is 0x7ff9a802cd10
request 0/2 instance is 0x7ff9b46f00c0
request 1/2 instance is 0x7ff9b46f00c0
[alkione:00000] *** An error occurred in MPI_Waitall
[alkione:00000] *** reported by process [1046151169,1]
[alkione:00000] *** on communicator MPI_COMM_WORLD
[alkione:00000] *** MPI_ERR_REQUEST: invalid request
[alkione:00000] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
[alkione:00000] ***    and MPI will try to terminate your MPI job as well)

Thanks!

[chhu]

Also, one even with just 2 ranks:

mpirun -n 2 ./gflops
MPI Enabled, node 0 of 2
Output is [min of single rank, max of single rank, TOTAL]
This benchmark does not account for MPI overhead. A large dot product is performed that quickly evades CPU cache.
Use: ./gflops <MByte to test per rank> <N loops>
Using default: 200MB / rank, 30 loops.
Dot product with 2 13107200 double arrays per rank
Single reference for speedup:  1.52 GFLOPs
Run 1: [ 1.03,  1.04,  2.07] GFLOP/s (FMA is 1 FLOP), [ 15.4,  15.4,  30.8] GByte/s. Speedup: 1.362
Run 2: [ 1.03,  1.04,  2.07] GFLOP/s (FMA is 1 FLOP), [ 15.4,  15.5,  30.8] GByte/s. Speedup: 1.362
Run 3: [ 1.03,  1.04,  2.07] GFLOP/s (FMA is 1 FLOP), [ 15.4,  15.5,  30.9] GByte/s. Speedup: 1.365
Run 4: [ 1.03,  1.04,  2.07] GFLOP/s (FMA is 1 FLOP), [ 15.4,  15.4,  30.8] GByte/s. Speedup: 1.361
Run 5: [ 1.03,  1.04,  2.07] GFLOP/s (FMA is 1 FLOP), [ 15.4,  15.5,  30.9] GByte/s. Speedup: 1.366
Run 6: [ 1.03,  1.04,  2.07] GFLOP/s (FMA is 1 FLOP), [ 15.4,  15.5,  30.9] GByte/s. Speedup: 1.365
Run 7: [ 1.03,  1.04,  2.07] GFLOP/s (FMA is 1 FLOP), [ 15.4,  15.4,  30.8] GByte/s. Speedup: 1.361
Run 8: [ 1.03,  1.04,  2.07] GFLOP/s (FMA is 1 FLOP), [ 15.4,  15.5,  30.9] GByte/s. Speedup: 1.366
Run 9: [ 1.03,  1.04,  2.07] GFLOP/s (FMA is 1 FLOP), [ 15.4,  15.5,  30.9] GByte/s. Speedup: 1.364
Run 10: [ 1.03,  1.04,  2.07] GFLOP/s (FMA is 1 FLOP), [ 15.4,  15.5,  30.9] GByte/s. Speedup: 1.365
Run 11: [ 1.03,  1.04,  2.07] GFLOP/s (FMA is 1 FLOP), [ 15.4,  15.4,  30.8] GByte/s. Speedup: 1.361
Run 12: [ 1.03,  1.04,  2.07] GFLOP/s (FMA is 1 FLOP), [ 15.4,  15.5,  30.8] GByte/s. Speedup: 1.363
Run 13: [ 1.02,  1.02,  2.04] GFLOP/s (FMA is 1 FLOP), [ 15.2,  15.3,  30.4] GByte/s. Speedup: 1.345
Run 14: [ 1.03,  1.04,  2.07] GFLOP/s (FMA is 1 FLOP), [ 15.4,  15.5,  30.9] GByte/s. Speedup: 1.364
Run 15: [ 1.04,  1.04,  2.07] GFLOP/s (FMA is 1 FLOP), [ 15.4,  15.5,  30.9] GByte/s. Speedup: 1.366
Run 16: [ 1.03,  1.04,  2.07] GFLOP/s (FMA is 1 FLOP), [ 15.4,  15.5,  30.9] GByte/s. Speedup: 1.365
Run 17: [ 1.03,  1.04,  2.07] GFLOP/s (FMA is 1 FLOP), [ 15.4,  15.5,  30.9] GByte/s. Speedup: 1.365
Run 18: [ 1.04,  1.04,  2.08] GFLOP/s (FMA is 1 FLOP), [ 15.4,  15.5,  30.9] GByte/s. Speedup: 1.367
Run 19: [ 1.03,  1.04,  2.07] GFLOP/s (FMA is 1 FLOP), [ 15.4,  15.5,  30.9] GByte/s. Speedup: 1.364
Run 20: [ 1.03,  1.04,  2.08] GFLOP/s (FMA is 1 FLOP), [ 15.4,  15.5,    31] GByte/s. Speedup: 1.368
Run 21: [ 1.03,  1.04,  2.07] GFLOP/s (FMA is 1 FLOP), [ 15.4,  15.5,  30.9] GByte/s. Speedup: 1.366
Run 22: [ 1.03,  1.04,  2.07] GFLOP/s (FMA is 1 FLOP), [ 15.4,  15.5,  30.9] GByte/s. Speedup: 1.366
Run 23: [ 1.03,  1.03,  2.06] GFLOP/s (FMA is 1 FLOP), [ 15.3,  15.4,  30.7] GByte/s. Speedup: 1.356
Run 24: [ 1.03,  1.04,  2.07] GFLOP/s (FMA is 1 FLOP), [ 15.3,  15.5,  30.8] GByte/s. Speedup: 1.361
Run 25: [ 1.03,  1.04,  2.07] GFLOP/s (FMA is 1 FLOP), [ 15.4,  15.5,  30.9] GByte/s. Speedup: 1.365
Run 26: [ 1.03,  1.04,  2.07] GFLOP/s (FMA is 1 FLOP), [ 15.4,  15.4,  30.8] GByte/s. Speedup: 1.362
Run 27: [ 1.03,  1.04,  2.07] GFLOP/s (FMA is 1 FLOP), [ 15.4,  15.5,  30.8] GByte/s. Speedup: 1.363
Run 28: [ 1.03,  1.04,  2.07] GFLOP/s (FMA is 1 FLOP), [ 15.4,  15.4,  30.8] GByte/s. Speedup: 1.362
Run 29: [ 1.03,  1.04,  2.07] GFLOP/s (FMA is 1 FLOP), [ 15.4,  15.4,  30.8] GByte/s. Speedup: 1.361
requests from different instances: 0xafc330 != (nil)
MPI_COMM_WORLD instance is 0xafc330
request 0/2 instance is 0x7f973e9730c0
request 1/2 instance is 0x7f973e9730c0
Run 30: [ 1.03,  1.04,  2.07] GFLOP/s (FMA is 1 FLOP), [ 15.4,  15.4,  30.8] GByte/s. Speedup: 1.36
Peak performance: 2.0773 GFLOP/s.
Average performance: 2.06951 GFLOP/s.

MPI-Latency and Bandwidth Test
Kind				bytes	time (sec)	Rate (MB/sec)
[alkione:00000] *** An error occurred in MPI_Waitall
[alkione:00000] *** reported by process [181469185,1]
[alkione:00000] *** on communicator MPI_COMM_WORLD
[alkione:00000] *** MPI_ERR_REQUEST: invalid request
[alkione:00000] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
[alkione:00000] ***    and MPI will try to terminate your MPI job as well)

Also, on an unrelated matter, the default mapping seems to have changed in MPI5, I have to explicitly tell --map-by numa to get the expected performance (if I don't exhaust all cores).

[ggouaillardet]

Thanks! I cannot make sense of it yet..
could you please revert the previous patch and apply this one instead?

diff --git a/ompi/mpi/c/waitall.c b/ompi/mpi/c/waitall.c
index ce2353dfc9..b95e6a0029 100644
--- a/ompi/mpi/c/waitall.c
+++ b/ompi/mpi/c/waitall.c
@@ -59,6 +59,7 @@ int MPI_Waitall(int count, MPI_Request requests[], MPI_Status statuses[])
         int rc = MPI_SUCCESS;
         OMPI_ERR_INIT_FINALIZE(FUNC_NAME);
         if( (NULL == requests) && (0 != count) ) {
+            fprintf(stderr, "MPI_Waitall(%d, NULL, ...)\n", count);
             rc = MPI_ERR_REQUEST;
         } else {
             if(!ompi_request_check_same_instance(requests, count) ) {
diff --git a/ompi/request/request.c b/ompi/request/request.c
index 843d9ae95f..03ebc47c94 100644
--- a/ompi/request/request.c
+++ b/ompi/request/request.c
@@ -31,6 +31,8 @@
 
 #include "ompi_config.h"
 
+#include <poll.h>
+
 #include "ompi/communicator/communicator.h"
 #include "opal/class/opal_object.h"
 #include "ompi/request/request.h"
@@ -267,8 +269,30 @@ bool ompi_request_check_same_instance(ompi_request_t** requests,
             base_instance = req->req_mpi_object.comm->instance;
             continue;
         }
-        if(base_instance != req->req_mpi_object.comm->instance)
+        if(base_instance != req->req_mpi_object.comm->instance) {
+            fprintf(stderr, "requests from different instances: %p != %p\n", base_instance, req->req_mpi_object.comm->instance);
+            fprintf(stderr, "MPI_COMM_WORLD instance is %p\n", ompi_mpi_comm_world.comm.instance);
+            for (int i=0; i<count; i++) {
+                if (OMPI_REQUEST_NULL == req->req_type) {
+                    fprintf(stderr, "request %d/%d is OMPI_REQUEST_NULL\n", i, count);
+                } else if (OMPI_REQUEST_PML != req->req_type) {
+                    fprintf(stderr, "request %d/%d is not OMPI_REQUEST_PML\n", i, count);
+                } else {
+                    fprintf(stderr, "request %d/%d instance is %p on comm %p\n", i, count, requests[i]->req_mpi_object.comm->instance, requests[i]->req_mpi_object.comm);
+                }
+            }
+            poll(NULL, 0, 1000);
+            for (int i=0; i<count; i++) {
+                if (OMPI_REQUEST_NULL == req->req_type) {
+                    fprintf(stderr, "request %d/%d is OMPI_REQUEST_NULL\n", i, count);
+                } else if (OMPI_REQUEST_PML != req->req_type) {
+                    fprintf(stderr, "request %d/%d is not OMPI_REQUEST_PML\n", i, count);
+                } else {
+                    fprintf(stderr, "request %d/%d instance is %p on comm %p\n", i, count, requests[i]->req_mpi_object.comm->instance, requests[i]->req_mpi_object.comm);
+                }
+            }
             return false;
+        }
     }
     return true;
 }

There are (oversimplified) two ways of doing communications within a node:

• use the UCX library to do most of the job: this is achieved when pml/ucx component is selected, which is the default if Infiniband is found
• use the generic pml/ob1 component is selected (that then will use the btl/sm component)

The former is relatively recent and relies on a bug-free UCX library, but the latter has been available and tested for years.

[chhu]

Hmm. So the bug probably comes from ucx...

MPI-Latency and Bandwidth Test
Kind				bytes	time (sec)	Rate (MB/sec)
requests from different instances: 0x15bfaf0 != (nil)
MPI_COMM_WORLD instance is 0x15bfaf0
request 0/2 instance is 0x15bfaf0 on comm 0x7feb6ccf5080
request 1/2 instance is (nil) on comm 0x7feb6ccf4c80
request 0/2 instance is 0x15bfaf0 on comm 0x7feb6ccf5080
request 1/2 instance is (nil) on comm 0x7feb6ccf4c80
[alkione:00000] *** An error occurred in MPI_Waitall
[alkione:00000] *** reported by process [1020788737,1]
[alkione:00000] *** on communicator MPI_COMM_WORLD
[alkione:00000] *** MPI_ERR_REQUEST: invalid request
[alkione:00000] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
[alkione:00000] ***    and MPI will try to terminate your MPI job as well)

[ggouaillardet]

Well, at this stage all we can observe is UCX is involved.
That being said, I am very puzzled to see the requests are on different communicators(!)

Not quite sure whether UCX is doing something unexpected, or if the request was corrupted.

What if you revert and apply the following patch?

diff --git a/ompi/mpi/c/waitall.c b/ompi/mpi/c/waitall.c
index ce2353dfc9..b95e6a0029 100644
--- a/ompi/mpi/c/waitall.c
+++ b/ompi/mpi/c/waitall.c
@@ -59,6 +59,7 @@ int MPI_Waitall(int count, MPI_Request requests[], MPI_Status statuses[])
         int rc = MPI_SUCCESS;
         OMPI_ERR_INIT_FINALIZE(FUNC_NAME);
         if( (NULL == requests) && (0 != count) ) {
+            fprintf(stderr, "MPI_Waitall(%d, NULL, ...)\n", count);
             rc = MPI_ERR_REQUEST;
         } else {
             if(!ompi_request_check_same_instance(requests, count) ) {
diff --git a/ompi/request/request.c b/ompi/request/request.c
index 843d9ae95f..88d82358b4 100644
--- a/ompi/request/request.c
+++ b/ompi/request/request.c
@@ -31,6 +31,8 @@
 
 #include "ompi_config.h"
 
+#include <poll.h>
+
 #include "ompi/communicator/communicator.h"
 #include "opal/class/opal_object.h"
 #include "ompi/request/request.h"
@@ -267,8 +269,42 @@ bool ompi_request_check_same_instance(ompi_request_t** requests,
             base_instance = req->req_mpi_object.comm->instance;
             continue;
         }
-        if(base_instance != req->req_mpi_object.comm->instance)
+        if(base_instance != req->req_mpi_object.comm->instance) {
+            fprintf(stderr, "requests from different instances: %p != %p\n", base_instance, req->req_mpi_object.comm->instance);
+            fprintf(stderr, "MPI_COMM_WORLD instance is %p\n", ompi_mpi_comm_world.comm.instance);
+            fprintf(stderr, "MPI_COMM_WORLD is %p\n", &ompi_mpi_comm_world.comm);
+            fprintf(stderr, "MPI_COMM_NULL is %p\n", &ompi_mpi_comm_null.comm);
+            fprintf(stderr, "MPI_COMM_SELF is %p\n", &ompi_mpi_comm_self.comm);
+            for (int i=0; i<count; i++) {
+                if (OMPI_REQUEST_NULL == req->req_type) {
+                    fprintf(stderr, "request %d/%d is OMPI_REQUEST_NULL\n", i, count);
+                } else if (OMPI_REQUEST_PML != req->req_type) {
+                    fprintf(stderr, "request %d/%d is not OMPI_REQUEST_PML\n", i, count);
+                } else {
+                    fprintf(stderr, "request %d/%d instance is %p on comm %p\n", i, count, requests[i]->req_mpi_object.comm->instance, requests[i]->req_mpi_object.comm);
+                }
+            }
+            poll(NULL, 0, 1000);
+            for (int i=0; i<count; i++) {
+                if (OMPI_REQUEST_NULL == req->req_type) {
+                    fprintf(stderr, "request %d/%d is OMPI_REQUEST_NULL\n", i, count);
+                } else if (OMPI_REQUEST_PML != req->req_type) {
+                    fprintf(stderr, "request %d/%d is not OMPI_REQUEST_PML\n", i, count);
+                } else {
+                    fprintf(stderr, "request %d/%d instance is %p on comm %p\n", i, count, requests[i]->req_mpi_object.comm->instance, requests[i]->req_mpi_object.comm);
+                }
+            }
+            for (int i=0; i<count; i++) {
+                if (OMPI_REQUEST_NULL == req->req_type) {
+                    fprintf(stderr, "request %d/%d is OMPI_REQUEST_NULL\n", i, count);
+                } else if (OMPI_REQUEST_PML != req->req_type) {
+                    fprintf(stderr, "request %d/%d is not OMPI_REQUEST_PML\n", i, count);
+                } else {
+                    fprintf(stderr, "request %d/%d instance is %p on comm %p (%d:%s)\n", i, count, requests[i]->req_mpi_object.comm->instance, requests[i]->req_mpi_object.comm, requests[i]->req_mpi_object.comm->super.s_base.obj_reference_count, requests[i]->req_mpi_object.comm->c_name);
+                }
+            }
             return false;
+        }
     }
     return true;
 }

[chhu]

Here you go.
I have to leave for today. Thanks for looking into this!

MPI-Latency and Bandwidth Test
Kind				bytes	time (sec)	Rate (MB/sec)
requests from different instances: 0x1e42420 != (nil)
MPI_COMM_WORLD instance is 0x1e42420
MPI_COMM_WORLD is 0x7f03a1cb2080
MPI_COMM_NULL is 0x7f03a1cb1c80
MPI_COMM_SELF is 0x7f03a1cb1e80
request 0/2 instance is 0x1e42420 on comm 0x7f03a1cb2080
request 1/2 instance is (nil) on comm 0x7f03a1cb1c80
request 0/2 instance is 0x1e42420 on comm 0x7f03a1cb2080
request 1/2 instance is (nil) on comm 0x7f03a1cb1c80
request 0/2 instance is 0x1e42420 on comm 0x7f03a1cb2080 (1:MPI_COMM_WORLD)
request 1/2 instance is (nil) on comm 0x7f03a1cb1c80 (3:MPI_COMM_NULL)
[alkione:00000] *** An error occurred in MPI_Waitall
[alkione:00000] *** reported by process [996212737,1]
[alkione:00000] *** on communicator MPI_COMM_WORLD
[alkione:00000] *** MPI_ERR_REQUEST: invalid request
[alkione:00000] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
[alkione:00000] ***    and MPI will try to terminate your MPI job as well)

[ggouaillardet]

Great, thanks again, I now have a hint of what could be going wrong.

[ggouaillardet]

I issued #12944, feel free to give it a try when you get some time.

[chhu]

Confirmed, works like a charm. Thanks!!