Cancel queued Publish RPCs if IDONTWANT arrives in the meantime

[raulk]

See libp2p/rust-libp2p#5751 and sigp/rust-libp2p#570.

[MarcoPolo]

We effectively do this already here by checking if the message is unwanted right before sending it.

[raulk]

sendRPC only queues the RPC for sending. The actual sending happens async via a dispatch goroutine a few layers down the stack. Between now and then we could've received an IDONTWANT. That's the cancellation I was referring to here. Some useful breadcrumbs:

• doSendRPC
• rpcQueue#push
• push
• priorityQueue#NormalPush

One way to approach this is by adding an RPC#Cancelled() bool that is backed by a closure supplied at RPC creation time that queries the gs.unwanted map, but we'd need to guard that with a lock, break the concurrency model, and introduce contention which ain't good. There are more ways.

[MarcoPolo]

This is likely a difference of <1ms for a machine not in high cpu contention. Is the juice worth the squeeze? We can certainly try.

[MarcoPolo]

I'd be curious to look at our trace metrics and see the time differences between when we send the rpc (in doSendRPC) and when we receive a IDONTWANT for that rpc, and how many times these are within 1 ms of each other.

[raulk]

I don't think this is CPU-bound? I was thinking it would be IO bound by the multiplexer or transport congestion control (i.e. if the Yamux window is full), which can happen more frequently under bandwidth saturation (especially under bursty loads).

[MarcoPolo]

ah right, if the transport backpressures on stream writes. That seems more likely 👍