Support a handler for checking connection status using Ping frame in HTTP/2 #3612

raccoonback · 2025-02-02T03:57:11Z

Description

In some cases, HTTP/2 connections may remain open even when they are no longer functional. By introducing a periodic Ping frame health check, we can proactively detect and close unhealthy connections.

The Http2ConnectionLivenessHandler schedules and sends Ping frames at a user-defined interval to verify the connection's liveness. If an acknowledgment (ACK) is not received within the specified time, the connection is considered unhealthy and is closed automatically.

However, if other frames are actively being sent or received, the scheduler does not send a Ping frame. This is because the server may delay ACK responses for various reasons. To prevent unnecessary connection termination, Ping frames are only sent when no read or write activity is detected.

Additionally, a configurable retry threshold for Ping frame transmission has been introduced. If a Ping frame fails to receive an ACK response, it will be retried up to the specified threshold before considering the connection unhealthy. This allows fine-tuning of the failure detection mechanism, balancing between aggressive failure detection and avoiding premature disconnections.

Scheduler Flow

Ping ACK Flow

Key Changes

Added Http2ConnectionLivenessHandler handler to check connection health with Ping frames at a configurable interval.
Introduced a retry threshold setting to limit the number of Ping transmission attempts before marking the connection as unhealthy.
Specify the ping frame ack allowable range through Http2SettingsSpec.

HttpClient.create()
	.protocol(HttpProtocol.H2)
	.secure()            
	.http2Settings(
		 builder -> builder.pingAckTimeout(Duration.ofMillis(600))  // Max wait time for ACK
			.pingScheduleInterval(Duration.ofMillis(300))  // Interval for sending PING frames
			.pingAckDropThreshold(2)  // Maximum retries before considering the connection dead
         );

Related Issue

Fix HTTP/2 PING frame handling #3301

raccoonback · 2025-02-08T05:52:25Z

reactor-netty-http/src/main/java/reactor/netty/http/Http2SettingsSpec.java

+		 * @param pingAckTimeout the interval in between consecutive PING frames
+		 *                       and ACK responses. Must be a positive value.
+		 */
+		default Builder pingAckTimeout(Duration pingAckTimeout) {


Sets the interval for sending HTTP/2 PING frames and receiving ACK responses.

raccoonback · 2025-02-08T05:52:38Z

reactor-netty-http/src/main/java/reactor/netty/http/Http2SettingsSpec.java

+		default Builder pingScheduleInterval(Duration pingScheduleInterval) {
+			return this;
+		}


Sets the execution interval for the scheduler that sends HTTP/2 PING frames

raccoonback · 2025-02-08T05:52:56Z

reactor-netty-http/src/main/java/reactor/netty/http/Http2SettingsSpec.java

+		default Builder pingAckDropThreshold(Integer pingAckDropThreshold) {
+			return this;
+		}


Sets the threshold for retrying HTTP/2 PING frame transmissions. (default. 0)

raccoonback · 2025-02-08T05:53:44Z

reactor-netty-http/src/main/java/reactor/netty/http/Http2SettingsSpec.java

+		/**
+		 * Sets the interval for sending HTTP/2 PING frames and receiving ACK responses.
+		 *
+		 * <p>
+		 * This method configures the time interval at which PING frames are sent to the peer.
+		 * The interval should be chosen carefully to balance between detecting connection issues
+		 * and minimizing unnecessary network traffic.
+		 * </p>
+		 *
+		 * <p>
+		 * If the interval is set too short, it may cause excessive network overhead.
+		 * If set too long, connection failures may not be detected in a timely manner.
+		 * </p>
+		 *
+		 * @param pingAckTimeout the interval in between consecutive PING frames
+		 *                       and ACK responses. Must be a positive value.
+		 */
+		default Builder pingAckTimeout(Duration pingAckTimeout) {
+			return this;
+		}
+
+		/**
+		 * Sets the execution interval for the scheduler that sends HTTP/2 PING frames
+		 * and periodically checks for ACK responses.
+		 *
+		 * <p>
+		 * This method configures the time interval at which the scheduler runs
+		 * to send PING frames and verify if ACK responses are received within
+		 * the expected timeframe.
+		 * Proper tuning of this interval helps in detecting connection issues
+		 * while avoiding unnecessary network overhead.
+		 * </p>
+		 *
+		 * <p>
+		 * If the interval is too short, it may increase network and CPU usage.
+		 * Conversely, setting it too long may delay the detection of connection failures.
+		 * </p>
+		 *
+		 * @param pingScheduleInterval the interval in at which the scheduler executes.
+		 *                       Must be a positive value.
+		 */
+		default Builder pingScheduleInterval(Duration pingScheduleInterval) {
+			return this;
+		}
+
+		/**
+		 * Sets the threshold for retrying HTTP/2 PING frame transmissions.
+		 *
+		 * <p>
+		 * This method defines the maximum number of attempts to send a PING frame
+		 * before considering the connection as unresponsive.
+		 * If the threshold is exceeded without receiving an ACK response,
+		 * the connection may be closed or marked as unhealthy.
+		 * </p>
+		 *
+		 * <p>
+		 * A lower threshold can detect connection failures more quickly but may lead
+		 * to premature disconnections. Conversely, a higher threshold allows more retries
+		 * but may delay failure detection.
+		 * </p>
+		 *
+		 * <p>
+		 * If this value is not specified, it defaults to 0, meaning only one attempt to send a PING frame is made without retries.
+		 * </p>
+		 *
+		 * @param pingAckDropThreshold the maximum number of PING transmission attempts.
+		 *                 Must be a positive integer.
+		 *                 The default value is 0, meaning no retries will occur and only one PING frame will be sent.
+		 */
+		default Builder pingAckDropThreshold(Integer pingAckDropThreshold) {
+			return this;
+		}


It is declared as a default method for backward compatibility.

raccoonback · 2025-02-08T05:55:15Z

reactor-netty-http/src/main/java/reactor/netty/http/client/Http2ConnectionLivenessHandler.java

+	private final long pingAckTimeoutNanos;
+	private final long pingScheduleIntervalNanos;
+	private final int pingAckDropThreshold;


This is the configuration value provided by the user.

raccoonback · 2025-02-08T06:22:04Z

reactor-netty-http/src/main/java/reactor/netty/http/client/Http2ConnectionLivenessHandler.java

+		private boolean isOutOfTimeRange() {
+			return pingAckTimeoutNanos < Math.abs(lastReceivedPingTime - lastSendingPingTime);
+		}


It determines whether the Ping ACK was received within pingAckTimeoutNanos.

raccoonback · 2025-02-08T06:24:06Z

reactor-netty-http/src/main/java/reactor/netty/http/client/Http2ConnectionLivenessHandler.java

+	private class PingWriteListener implements ChannelFutureListener {
+
+		@Override
+		public void operationComplete(ChannelFuture future) throws Exception {
+			if (future.isSuccess()) {
+				if (log.isDebugEnabled()) {
+					log.debug("Wrote PING frame to {} channel.", future.channel());
+				}
+
+				isPingAckPending = true;
+				lastSendingPingTime = System.nanoTime();
+			}
+			else if (log.isDebugEnabled()) {
+				log.debug("Failed to wrote PING frame to {} channel.", future.channel());
+			}
+		}
+	}


If the new Ping frame is successfully sent, the status is changed to 'ping awaiting', and lastSendingPingTime is updated to the current time.

raccoonback · 2025-02-08T06:25:03Z

reactor-netty-http/src/main/java/reactor/netty/http/client/HttpClientConfig.java

+		 .addBefore(NettyPipeline.ReactiveBridge, NettyPipeline.H2LivenessHandler,
+				 new Http2ConnectionLivenessHandler(codec.encoder(), pingAckTimeout, pingScheduleInterval, pingAckDropThreshold))


configure Http2ConnectionLivenessHandler.

raccoonback · 2025-02-08T06:27:08Z

...y-http/src/test/java/reactor/netty/resources/PooledConnectionProviderDefaultMetricsTest.java

+		REGISTRY.clear();
+


Although not related to the purpose of this PR, handling has been implemented for failing tests.
To prevent interference with other tests, the metrics are initialized.

raccoonback · 2025-02-08T06:29:41Z

reactor-netty-http/src/test/java/reactor/netty/http/client/HttpClientTest.java

+			Mockito.verify(spyLogger, times(0))
+					.warn(Mockito.eq("Connection pool creation limit exceeded: {} pools created, maximum expected is {}"),
+							Mockito.eq(2), Mockito.eq(1));


Although not related to the purpose of this PR, handling has been implemented for failing tests.
In PooledConnectionProvider, the connectionPoolCount is handled atomically to ensure that the log warning log.warn("Connection pool creation limit exceeded: {} pools created, maximum expected is {}") is not called multiple times.
As a result, the test outcomes have been adjusted.

violetagg · 2025-02-11T17:09:30Z

@raccoonback Can you rebase against branch 1.2.x and switch the target branch for the PR to be 1.2.x. We branched and now in main it is the next development 1.3.x while we want this feature for version 1.2.x

raccoonback · 2025-02-11T23:50:11Z

@violetagg
Hello!
I have rebased the branch onto 1.2.x and changed the target branch of the PR to 1.2.x.

Signed-off-by: raccoonback <[email protected]>

… in HTTP/2 Signed-off-by: raccoonback <[email protected]>

Signed-off-by: raccoonback <[email protected]>

…e ping interval - ref. https://github.com/reactor/reactor-netty/actions/runs/13095356929/job/36537031690?pr=3612 Signed-off-by: raccoonback <[email protected]>

Signed-off-by: raccoonback <[email protected]>

- Added a method to configure the execution interval of the scheduler that sends HTTP/2 PING frames and periodically checks for ACK responses - Introduced a retry threshold setting to limit the number of PING transmission attempts before considering the connection as unresponsive - Default values: - Scheduler interval must be explicitly set - Retry threshold defaults to 0 (no retries, only one PING attempt) Signed-off-by: raccoonback <[email protected]>

violetagg · 2025-02-17T15:24:59Z

@raccoonback I'll review this PR later this week

violetagg · 2025-02-20T14:18:38Z

@raccoonback Nice idea! I think that we can reuse our current reactor.netty.http.server.IdleTimeoutHandler (we need to check whether the current position in the pipeline will be ok or whether we need to move it) and extend it with the HTTP/2 functionality ACK requirement. At the moment it handles idle timeout for HTTP/2 and H2C on the server. The more ChannelHandler you have in the pipeline the worse is the performance so we always try to keep the ChannelHandlers number to the minimum if possible. Also all methods in reactor.netty.http.server.IdleTimeoutHandler are invoked on the event loop which guarantees that the functionality is thread-safe.

I think this feature is interesting for both server and client, similar to TCP keep-alive that we support for both of them.

raccoonback · 2025-02-20T23:36:24Z

@violetagg
Hello, and thanks for your suggestion! 😃
I have a few questions for clarification.

You suggested reusing reactor.netty.http.server.IdleTimeoutHandler.
Currently, the implementation works by scheduling a PING frame transmission when there is no read/write activity, then checking whether an ACK is received within the allowed time. (It also supports retries up to a certain threshold.)
'
I understand that you’re suggesting leveraging IdleTimeoutHandler to detect when both read and write operations are idle and incorporating it into this logic. Is that correct?
Specifically, do you mean modifying the handler so that when both read and write operations are idle, a PING frame is sent, and if no ACK is received within the retry threshold, the connection is closed?
'
If I’ve misunderstood, could you clarify what exactly you mean by "reuse"? 😮
Can I understand that you are proposing to extend this feature not only on the client side, but also on the server side by IdleTimeoutHandler?
- I thought that this Issue was intended to introduce PING frame support only for the client-side at first.

Looking forward to your feedback! 😊

violetagg · 2025-02-21T10:17:18Z

@violetagg Hello, and thanks for your suggestion! 😃 I have a few questions for clarification.

You suggested reusing reactor.netty.http.server.IdleTimeoutHandler.
Currently, the implementation works by scheduling a PING frame transmission when there is no read/write activity, then checking whether an ACK is received within the allowed time. (It also supports retries up to a certain threshold.)
'
I understand that you’re suggesting leveraging IdleTimeoutHandler to detect when both read and write operations are idle and incorporating it into this logic. Is that correct?

Yes
Currently we use it only for checking that there is no read, because we add this handler between the requests and we are sure there is no write operation. However this handler extends io.netty.handler.timeout.IdleStateHandler and for HTTP/2 we can configure it to check for both read and write.

Specifically, do you mean modifying the handler so that when both read and write operations are idle, a PING frame is sent, and if no ACK is received within the retry threshold, the connection is closed?

Yes
When we receive channelIdle we can add the logic for ACK

'
If I’ve misunderstood, could you clarify what exactly you mean by "reuse"? 😮
2. Can I understand that you are proposing to extend this feature not only on the client side, but also on the server side by IdleTimeoutHandler?

Yes

I thought that this Issue was intended to introduce PING frame support only for the client-side at first.

Looking forward to your feedback! 😊

We may need to move reactor.netty.http.server.IdleTimeoutHandler to the reactor.netty.http package so that it can be used by both server and client.

I think that this feature is interesting for both client and server. Wdyt?

raccoonback · 2025-02-21T23:25:23Z

@violetagg

Thank you for the detailed explanation!
I will update reactor.netty.http.server.IdleTimeoutHandler to check the Ping ACK internally when usingHTTP/2 and H2C.

raccoonback marked this pull request as ready for review February 2, 2025 08:17

raccoonback force-pushed the issue-3301 branch 9 times, most recently from 171dfc5 to 202d036 Compare February 4, 2025 00:11

raccoonback marked this pull request as draft February 5, 2025 16:36

raccoonback force-pushed the issue-3301 branch from 18196bd to 231b27d Compare February 7, 2025 16:34

raccoonback marked this pull request as ready for review February 7, 2025 17:12

raccoonback commented Feb 8, 2025

View reviewed changes

violetagg added the type/enhancement A general enhancement label Feb 10, 2025

violetagg added this to the 1.2.4 milestone Feb 10, 2025

raccoonback force-pushed the issue-3301 branch from 231b27d to a06c842 Compare February 11, 2025 23:47

raccoonback changed the base branch from main to 1.2.x February 11, 2025 23:47

raccoonback added 10 commits February 17, 2025 23:47

Support configuring the Ping frame interval in the HTTP/2 protocol

3193c8f

Signed-off-by: raccoonback <[email protected]>

Add a handler to check connection status via Ping frame health checks…

594a914

… in HTTP/2 Signed-off-by: raccoonback <[email protected]>

Update doc related with HTTP/2

94301bd

Signed-off-by: raccoonback <[email protected]>

Ensure backward compatibility while adding support for configuring th…

b046066

…e ping interval - ref. https://github.com/reactor/reactor-netty/actions/runs/13095356929/job/36537031690?pr=3612 Signed-off-by: raccoonback <[email protected]>

Fix unnecessary warnings in test code

c6b9206

Signed-off-by: raccoonback <[email protected]>

Http2ChannelDuplexHandler -> ChannelDuplexHandler

09da411

Signed-off-by: raccoonback <[email protected]>

Reduce the scope of access control

868456a

Signed-off-by: raccoonback <[email protected]>

Fix broken test code

a31ddef

Signed-off-by: raccoonback <[email protected]>

Change comparison type for PING frame

9575177

Signed-off-by: raccoonback <[email protected]>

Fix broken test

c3d91fb

Signed-off-by: raccoonback <[email protected]>

raccoonback force-pushed the issue-3301 branch from a06c842 to 8c4cdbc Compare February 17, 2025 14:51

violetagg self-requested a review February 17, 2025 15:25

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support a handler for checking connection status using Ping frame in HTTP/2 #3612

Support a handler for checking connection status using Ping frame in HTTP/2 #3612

raccoonback commented Feb 2, 2025 •

edited

Loading

raccoonback Feb 8, 2025

raccoonback Feb 8, 2025

raccoonback Feb 8, 2025

raccoonback Feb 8, 2025

raccoonback Feb 8, 2025

raccoonback Feb 8, 2025

raccoonback Feb 8, 2025

raccoonback Feb 8, 2025

raccoonback Feb 8, 2025

raccoonback Feb 8, 2025

violetagg commented Feb 11, 2025

raccoonback commented Feb 11, 2025

violetagg commented Feb 17, 2025

violetagg commented Feb 20, 2025 •

edited

Loading

raccoonback commented Feb 20, 2025 •

edited

Loading

violetagg commented Feb 21, 2025 •

edited

Loading

raccoonback commented Feb 21, 2025

		.addBefore(NettyPipeline.ReactiveBridge, NettyPipeline.H2LivenessHandler,
		new Http2ConnectionLivenessHandler(codec.encoder(), pingAckTimeout, pingScheduleInterval, pingAckDropThreshold))

Support a handler for checking connection status using Ping frame in HTTP/2 #3612

Are you sure you want to change the base?

Support a handler for checking connection status using Ping frame in HTTP/2 #3612

Conversation

raccoonback commented Feb 2, 2025 • edited Loading

Description

Scheduler Flow

Ping ACK Flow

Key Changes

Related Issue

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

violetagg commented Feb 11, 2025

raccoonback commented Feb 11, 2025

violetagg commented Feb 17, 2025

violetagg commented Feb 20, 2025 • edited Loading

raccoonback commented Feb 20, 2025 • edited Loading

violetagg commented Feb 21, 2025 • edited Loading

raccoonback commented Feb 21, 2025

raccoonback commented Feb 2, 2025 •

edited

Loading

violetagg commented Feb 20, 2025 •

edited

Loading

raccoonback commented Feb 20, 2025 •

edited

Loading

violetagg commented Feb 21, 2025 •

edited

Loading