MPI_Comm_create_from_group works incorrectly when used with non-overlapping groups but same stringtag argument

[hppritcha]

Turns out the way MPI_Comm_create_from_group is using the PMIx_Group_construct method doesn't correctly handle the case where multiple MPI processes are invoking MPI_Comm_create_group using different, non-overlapping MPI groups, but the same stringtag argument.

The wording in the MPI-4 standard does not indicate this is incorrect:

If a nonempty group is specified, then all MPI processes in that group must call
the function and each of these MPI processes must provide the same arguments, including
a group that contains the same members with the same ordering, and identical stringtag
value.

@dalcini

[hppritcha]

I'm marking this as a blocker because its solution will likely involve changes to a parameter in the mpi.h.in header file - which is incorrect in any case for the maximum allowed stringtag length.

[rhc54]

So you are interpreting that statement as only requiring that all processes calling the function must pass the same stringtag, but not that the stringtag must be unique - yes? That strikes me as an oversight in the standard that needs to be addressed as it frankly makes no sense. Why "tag" a group if it isn't unique? What possible purpose could the tag serve in that case?

[hppritcha]

[bosilca]

The stringtag is not about identifying the group, because otherwise the MPI standard would be standardizing how external software would have to name their groups. Instead, the stringtag is an identifier for the operation on this particular group. To understands why this is necessary one must look at the API of call, and notice that this API does not have a base communicator, which means there is no dedicated communication channel for this operation, aka. that the only way the processes can order concurrent calls to this function is by uniquely identifying them with the tuple group and tag. Without taking both in account you will either prevent concurrent calls to overlapping groups, or unordered calls with the same group.

[hppritcha]

This goes back to the above figure a process sets. We have a blurb in the MPI-4 standard, chapter 11, that states

If two MPI processes get the same process set name, then the intersection of the two 
process sets shall either be the empty set or identcial to the union of the two process sets.

So the MPI://SELF process set is an example of the former in the above statement, so I could see how a programmer may interpret that to mean that there's no reason for the stringtag to be unique if the group handle argument to the MPI_Comm_create_from_group came from non-overlapping process sets.

Any rate, I plan to fix this by appending the vpid of the first proc in the group to the supplied stringtag before turning over to PMIx_Group_construct. Thank goodness we retained the verbiage about restrictions on the group argument concerning order of processes in the group.

[bosilca]

Thank goodness we retained the verbiage about restrictions on the group argument concerning order of processes in the group.

What would have been the order of processes in the new communicator otherwise ?

[rhc54]

The stringtag is not about identifying the group,

I see - thanks for clarifying!

Any rate, I plan to fix this by appending the vpid of the first proc in the group to the supplied stringtag

I grok what you are saying, but that still doesn't result in a unique PMIx group name since the proc might be involved in multiple MPI_Comm_create_from_group calls and there is no guarantee that the user will only put that proc once in the first position, is there?

[rhc54]

Couple of responses/questions, but I'll post them separately.

What would have been the order of processes in the new communicator otherwise ?

I have to agree with this question. If a collection of procs calls this function using arbitrary ordering, then it isn't clear to me what the resulting communicator "rank" should be for the participants. If we follow typical MPI logic, it would be the position of the proc in the provided array - but that would differ across the collection. So it would seem that the ordering must be the same if you want to result in a usable MPI communicator, yes?

[rhc54]

@hppritcha I wonder if we are getting caught in a confusion caused by overloading of the "group" term. Are you equating a given MPI communicator with a specific PMIx group? If so, that has some implications that might not be acceptable to the MPI community. Biggest difference is that we have unique string "names" for a group that helps us distinguish between them, even if they have the same membership. You lack that in your communicators and I'm not sure how you'd add it.

If two MPI processes get the same process set name, then the intersection of the two
process sets shall either be the empty set or identcial to the union of the two process sets.

If this refers or attempts to correlate with PMIx process sets, then that wouldn't be correct. Because of the dynamic process set support, a process (in PMIx parlance) doesn't belong to just one process set. So two processes could find themselves members of the same process set, but that set could be completely orthogonal to any other set to which they belong.

Again, it may be that the terminology unfortunately overlaps and causes confusion. I'm only commenting out of concern regarding what PMIx can and cannot support, wanting to ensure that we do support whatever you are attempting to do.

[hppritcha]

@rhc54 getting side tracked today I'll answer questions here tomorrow. we really are only using PMIx groups as a rendezvous mechanism not as something for tagging comms, groups, etc.

[rhc54]

Gotcha - will look forward to your answers when you have time. FWIW, I attempted to clear up some terminology confusion here (#10886 (comment)) that might be relevant in this topic as well.