BOLT-0008+BOLT-0007: consider adding QUIC transport

[Roasbeef]

Background

So recently in lnd we had an inadvertent regression in the performance of our gossip message processing (never made it to a release, was caught in an rc). A few release backs, we started to actually enforce ping responses. So if a peer didn't respond to a ping in time with a pong response, we'd disconnect them. Initially we didn't realize there was a perf regression, and we just saw that many peers were now failing the ping liveness check.

TCP Head of Line Blocking

What happened was that we started to dramatically limit some internal message buffering, combined with the dramatic regression in gossip ingestion pace, messages were processed more slowly, and would then sit in an internal queue much longer. Once that queue filed out, the main thread that read messages off the wire would block.

If this thread blocks, then eventually the TCP buffers would fill up, and the connected peer would in turn block themselves. So even though a peer was still processing messages, they could still fail our pick check if: the ping message was stuck behind a flood of gossip messages, or the had a pong response, but we hadn't processed it yet.

This phenomena is typically referred to as head-of-the-line-blocking, it can happen at the application level, but its also inherent to the way TCP works, as it implements a reliable in-order stream abstraction.

After a bit of wack-a-mole, we found the performance regression, and reverted it. However if you come up with some arbitrary numbers for: the size of the incoming gossip queue, the message arrival rate, and the processing latency -- it's easy to come up with a combo that leads to a ping/pong message being "stuck" behind other messages, even at the kernel/TCP level.

That got me thinking: what if we actually added a QUIC transport mode?

P2P QUIC Transports & Concurrent Streams

QUIC unlike TCP, is UDP based. It moves the congestion control algorithm into user space, whereas for TCP all that logic lives firmly in the kernel. Part of the rationale here is that user space can evolve more quickly than kernel space can, which can help stave off ossification and promote experimentation.

In a p2p setting like ours, one interesting component of QUIC is the concept of streams. A single connection can have multiple streams, each stream has flow control handled separately from other streams. This serves to effectively isolate the QoS of a given stream. If one stream is moving slowly, that doesn't block any of the other stream. In the LN setting, this means that for a given channel/connection, we could have a stream for: gossip messages, ping/pongs, channel state machine updates, concurrent splices, etc, etc.

A TCP connection is effectively a single logical stream/queue. If the other side stops ACK'ing your packets, then eventually you'll just stop and wait to send more. Here we see a manifestation of head-of-the-line-blocking. Imagine we have an HTLC that we're ready to send out to complete a payment circuit, as the stream is processed linearly, that update_add_htlc message could get stuck behind a flood of channel_update messages. This directly impacts perceived payment latency on the network.

If a QUIC transport were to be added, then for a given channel/connection, we could have a stream for: gossip messages, ping/pongs, channel state machine updates, concurrent splices, etc, etc. Returning to the scenario above, slow processing of gossip messages would necessarily block sending out pings or normal channel update messages, etc.

Other advantages include a fast 1-RTT (and even R-TT) connection handshake, more seamless connection migration (connection ID is the main identifier vs the normal TCP 4-tuple), easier hole punching, etc.

Sketch of Change Across BOLT 8 & BOLT 7

If we wanted to add an option for a QUIC transport, one obvious area that would need extensions is the node_announcement message as defined in BOLT 7. We'd want to add an ability to signal that the peer supported a QUIC transport.

One complication with QUIC, is that AFAICT, there's no way to run without TLS 1.3. Today we run over plain TCP, then layer the noise protocol on top. As a result, AFAICT we'd just have to layer BOLT 8 on top (double encryption) and come up with some sort of mock TLS cert/scheme.

We'd likely also add some suggestion to the spec re how an implementation should handle stream handling, etc.

Potential Roadblocks

After all these years (QUIC was first publicly announced in 2013, IETF RFC published in 2015 ish) first-class support amongst popular programming languages doesn't seem to be very wide spread. For example, Go has an implementation in the stdlib, but it's under the /x prefix, which means it's experimental. There's also quite a lot of the underlying protocol that hasn't been implemented yet in the package.

There's also the whole TLS 1.3 requirement. I haven't dug super deep, but it appears that TLS is pretty intertwined w/ the protocol. It's possbile that there're lower level libraries that let you use QUIC w/o the TLS layer (eg:run in an insecure/debug mode), but further investigation is needed. QUIC connection establishment has less RTTs then TCP, so we'd have less RTT overall even combining the Noise handshake overhead.

AFAICT, it should be possible to just use raw public keys in place of self signed certs.

[tnull]

I generally like QUIC, appreciate it's benefits, and even explored its potential usefulness for Bitcoin's P2P layer at some point.

Stream isolation would be nice to have, but other than that I'm not convinced that it would gain us much for Lightning, where connections are mostly long-lived and hence handshake latency isn't that important.

That is assuming we generally want to maintain how the Lightning network layer works (noise protocol, unstructured gossip broadcasting). From my perspective, a switch to QUIC would only become attractive if we'd actually consider dropping noise for TLS, and using it's 1-RTT/0-RTT feature as part of a new gossip protocol that wouldn't necessarily only propagate gossip to just your long-lived peers.

Apart from that, the fact that QUIC speaks UDP and its implementations live in userland has several drawbacks. For one, it's not as ubiquitously available on all platforms/languages. It also means that the burden of maintaining the implementation dependencies becomes responsibility of the Lightning implementation maintainer, which might introduce additional complexity into our code.

TLDR: Like QUIC, like exploring the idea, but am pretty skeptical of the cost vs. benefits here.

[yyforyongyu]

From my perspective, a switch to QUIC would only become attractive if we'd actually consider dropping noise for TLS

Think there's nQUIC, which is noise-based, tho not sure if there are out-of-box implementations.

For one, it's not as ubiquitously available on all platforms/languages.

It's widely supported based on this answer. Tho not sure how easy it is for us to run QUIC with noise. Plus 1/3 of the websites now already use HTTP/3.

Stream isolation may become critical if the network keeps growing - channel state msgs should be prioritized, it can be an issue if a revoke_and_ack is piped after tons of gossip msgs. And the connection migration used in QUIC can be nice for mobile users.

[TheBlueMatt]

lol. QUIC is a kinda-bad way to do multi-stream-one-congestion-control-state-machine, while rewriting all of TCP in the process just for funzies. Its also super duper targeted at the web, so not really all that useful outside of it. The multi-stream thing only accomplishes anything if you're doing super short-lived connections (like on the web), so its also not really relevant to lightning, if you want multi-stream, you can justo pen two TCP sockets. You'll get the same thing as QUIC but without any complexity :)

[cdecker]

While I find QUIC a fascinating experiment, I don't think it is suitable for our applications, since we have rather good control over message flows in the protocol, if you squint at it. CLN, and others I'm sure, do not queue gossip willy nilly, but rather it gets pulled in at a sustainable rate, such that the head of line blocking does not manifest. Gossip aside there aren't any big traffic producers, and using QUIC in this case would just shift the burden of muxing gossip and realtime messages to QUICs muxing and prioritization, i.e., giving away the control to a less knowledgeable component.

I shudder thinking about having to implement and maintain a QUIC implementation to be honest (io_loop being used in CLN makes that necessary).

[Roasbeef]

CLN, and others I'm sure, do not queue gossip willy nilly, but rather it gets pulled in at a sustainable rate, such that the head of line blocking does not manifest

Does CLN drop explicitly incoming gossip packets under any scenario?

IIUC, there's a sub-process (gossipd) that handles gossip message processing, and presumably there's another sub-process (I'll call that connectd, dunno if that's the actual name) that handles reading messages off the wire, and then decides where to send them. Under what circumstances does connectd block? Let's say bitcoind is held up for some reason (say, 10s of seconds) which blocks gossipd from being able validate a new channel announcement (ignoring the internal UTXO set for the sake of the example).

If the scenario is taken at face value, how doesn't head-of-the-line blocking arise? connectd is blocked as it's waiting for gossipd to process new messages coming in. There's effectively a single queue, at play here. Any potential HTLC/sig messages remain unacked from the sender's PoV, so it doesn't continue sending until those are ack'd at the TCP level. connectd won't pull any additional messages off the wire until gossipd starts to process them.

[tnull]

We briefly mentioned this on the recent call, but I'd like to raise the question again whether it would make sense to have multiple (dedicated) TCP connections to isolate streams by concern. For example, you could have dedicated TCP connections for channel messages, for gossip, for onion messages, and for custom messages.

A byproduct of this would be that you could hit some kind of error case that would prompt you to close the connection to the counterparty, without infringing on the other concerns. We came across that previously when implementing the LSP specs which use BOLT8 custom messages as their transport layer: there are cases where the counterparty would for example send us some messages we are unable to parse and we'd usually just like to close the connection on them. However, we can't do that because in an LSP setting you might have channels open that should remain operational and you can't just close the connection because of an application-layer error. Having a dedicated TCP stream for custom messages, would hence help isolate different concerns and handle them differently.

[cdecker]

CLN, and others I'm sure, do not queue gossip willy nilly, but rather it gets pulled in at a sustainable rate, such that the head of line blocking does not manifest

Does CLN drop explicitly incoming gossip packets under any scenario?

IIUC, there's a sub-process (gossipd) that handles gossip message processing, and presumably there's another sub-process (I'll call that connectd, dunno if that's the actual name) that handles reading messages off the wire, and then decides where to send them. Under what circumstances does connectd block? Let's say bitcoind is held up for some reason (say, 10s of seconds) which blocks gossipd from being able validate a new channel announcement (ignoring the internal UTXO set for the sake of the example).

You're looking at the receiving side, where indeed we drop packets if a downstream consumer such as gossipd is overwhelmed (and it's queue is filled). There's little you can do on the sending side of a stream, other than process packet after packet, you must load shed. However, head of line blocking is rather easily prevented on the sending side: gossip is essentially just pulling the next message, when there isn't something more important, from an append only log in our case. That means there is no pressure on the incoming side of that queue, and we can use the send-when-not-busy logic to intersperse gossip messages in a way that doesn't delay an eventually arriving urgent message too much.

[TheBlueMatt]

The one important thing to keep in mind on the sending side is its very easy to shove enough gossip into the system TCP buffer that you have a peer that's overwhelmed by them. On the LDK side we're quite conservative and only put a few gossip messages into the host TCP queue before we send a ping and make sure we get a pong back to make sure we're not drowning the peer. IMO all nodes should do this (it slows down gossip a bit, but who cares)

[Roasbeef]

we can use the send-when-not-busy logic to intersperse gossip messages in a way that doesn't delay an eventually arriving urgent message too much

Yep, we also add delays before broadcasting out batches of gossip messages.

You're looking at the receiving side
However, head of line blocking is rather easily prevented on the sending side: gossip is essentially just pulling the next message

Yeah, I'm focused on the receiving side. As if the system receiving blocks, then sending isn't possible, and any messages in the logical connection queue will just sit there. Even if the sender commits to say only sending 5 gossip messages in some interval, if the receiver is very slow (slow gossip processing), then it's starved from processing potentially urgent messages (HTLC forwarding).

I've observed behavior in the wild where CLN takes 20+ seconds to respond to a ping with a pong: ElementsProject/lightning#8308

If we had distinct streams for pings vs gossip, then slow gossip processing wouldn't delay the processing of bigs.