New vader SEGV possibly PR 5829

[gpaulsen]

@amckinstry commented on #5829

Original issue was: #5638

Since that was merged to all release branches, this is a blocker on all release branches.

Unfortunately we now see a related crash on other codes (lammps):

#0 0x0000000000000000 in ()
#1 0x00007f7a700f8a5f in mca_btl_vader_poll_handle_frag (hdr=0x7f7a6a1bf049, endpoint=endpoint@entry=0x55e02df87bd0) at btl_vader_component.c:603
#2 0x00007f7a700f8f83 in mca_btl_vader_check_fboxes () at btl_vader_fbox.h:225

#3 0x00007f7a700f8f83 in mca_btl_vader_component_progress () at btl_vader_component.c:702
#4 0x00007f7a828fb1cc in opal_progress () at runtime/opal_progress.c:228
#5 0x00007f7a8d816ebd in ompi_request_wait_completion (req=0x55e02df8d900) at ../ompi/request/request.h:413
#6 0x00007f7a8d816ebd in ompi_request_default_wait (req_ptr=0x7fffa70c3708, status=0x7fffa70c3710) at request/req_wait.c:42
#7 0x00007f7a8d86ec41 in ompi_coll_base_sendrecv_actual (sendbuf=sendbuf@entry=0x55e02df97de0, scount=scount@entry=1, sdatatype=sdatatype@entry=0x55e02cace1e0 <ompi_mpi_int>, dest=dest@entry=0, stag=stag@entry=-12, recvbuf=recvbuf@entry=0x7fffa70c3934, rcount=1, rdatatype=0x55e02cace1e0 <ompi_mpi_int>, source=0, rtag=-12, comm=0x55e02cacf700 <ompi_mpi_comm_world>, status=0x0) at base/coll_base_util.c:59

it looks like hdr->tag is invaliid, hence segdfault

[gpaulsen]

@amckinstry, can you please confirm that you have commit dfa8d3a in your master build that you saw this with? There were a few other vader related fixes that went in around #5829

[gpaulsen]

@amckinstry - @hjelmn asked what platforms this affects. If it only affects i386, we may not have a developer to fix this.

[jsquyres]

I don't agree with that sentiment. If we know it's broken on a platform, we need to make it right. We either fix it on that platform or we disable the functionality (i.e., vader fastboxes) on that platform. For core functionality like this, we can't just knowingly have a bad bug like this.

[hjelmn]

@jsquyres My time is at a premium. We don't run i386 so unless I am told to fix this at work I will not spend any time on it. Just the reality right now. Easy enough to disable fast boxes on i386. I can make a PR for that.