Enable user-level threading

[wrrobin]

Issue

Current OpenSHMEM standard does not allow having user-level threads from applications. Supporting user-level threading effectively in an OpenSHMEM library requires integration of of cooperative thread scheduling to prevent deadlock in scenarios where threads block or poll on updates to memory (e.g. in a call to shmem_wait or repeated calls to shmem_test) and to hide latency in scenarios where threads become blocked on communication calls (e.g. in a call to shmem_get).

Proposed Solution

Add API to register a callback that OpenSHMEM library can utilize to yield the current thread.

void shmem_register_yield(void (*yield_fn)());

[wrrobin]

Slides for 10/9 meeting:
shmem_ult_10_09_2020.pptx

[wrrobin]

Closing as not enough use-cases currently.