Skip to content

handle java.util.concurrent.RejectedExecutionException in 7.0.1 #6747

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
vicozohili opened this issue Dec 20, 2024 · 2 comments · Fixed by #6760
Closed

handle java.util.concurrent.RejectedExecutionException in 7.0.1 #6747

vicozohili opened this issue Dec 20, 2024 · 2 comments · Fixed by #6760
Assignees
@shawkins
Milestone
7.1.0

Comments

@vicozohili
Copy link

Describe the bug

Same issue reported in major version 6.13.3 fixed in 6.13.4 is also occurring 7.0.1
see #6278

Fabric8 Kubernetes Client version

7.0.1

Steps to reproduce

.

Expected behavior

.

Runtime

OpenShift

Kubernetes API Server version

1.25.3@latest

Environment

Linux

Fabric8 Kubernetes Client Logs

java.util.concurrent.RejectedExecutionException: Task io.fabric8.kubernetes.client.utils.internal.SerialExecutor$$Lambda$1805/0x00007facb09bf648@70ab95e9 rejected from java.util.concurrent.ThreadPoolExecutor@4b42b445[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4]
	at java.base/java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2065)
	at java.base/java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:833)
	at java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1365)
	at io.fabric8.kubernetes.client.utils.internal.SerialExecutor.scheduleNext(SerialExecutor.java:75)
	at io.fabric8.kubernetes.client.utils.internal.SerialExecutor.execute(SerialExecutor.java:69)
	at io.fabric8.kubernetes.client.dsl.internal.ExecWebSocketListener.onClose(ExecWebSocketListener.java:373)
	at io.fabric8.kubernetes.client.vertx.VertxWebSocket.lambda$init$2(VertxWebSocket.java:55)
	at io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:270)
	at io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:252)
	at io.vertx.core.http.impl.WebSocketImplBase.receiveFrame(WebSocketImplBase.java:560)
	at io.vertx.core.streams.impl.InboundBuffer.handleEvent(InboundBuffer.java:279)
	at io.vertx.core.streams.impl.InboundBuffer.write(InboundBuffer.java:157)
	at io.vertx.core.http.impl.WebSocketImplBase.handleFrame(WebSocketImplBase.java:492)
	at io.vertx.core.impl.ContextImpl.execute(ContextImpl.java:327)
	at io.vertx.core.impl.ContextImpl.execute(ContextImpl.java:307)
	at io.vertx.core.http.impl.Http1xConnectionBase.handleWsFrame(Http1xConnectionBase.java:66)
	at io.vertx.core.http.impl.Http1xClientConnection.handleMessage(Http1xClientConnection.java:778)
	at io.vertx.core.net.impl.ConnectionBase.read(ConnectionBase.java:159)
	at io.vertx.core.net.impl.VertxHandler.channelRead(VertxHandler.java:153)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1503)
	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1366)
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1415)
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:530)
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:469)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1357)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:868)
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:840)

Additional context

No response
@vicozohili
Copy link
Author

2024-12-20 14:00:05,989 [ERROR] io.vertx.core.impl.ContextImpl - Unhandled exception
java.util.concurrent.RejectedExecutionException: Task io.fabric8.kubernetes.client.utils.internal.SerialExecutor$$Lambda$1668/0x00007f6be08b70b8@258d056d rejected from java.util.concurrent.ThreadPoolExecutor@2c1e62de[Shutting down, pool size = 1, active threads = 0, queued tasks = 0, completed tasks = 1]
at java.base/java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2065)
at java.base/java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:833)
at java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1365)
at io.fabric8.kubernetes.client.utils.internal.SerialExecutor.scheduleNext(SerialExecutor.java:75)
at io.fabric8.kubernetes.client.utils.internal.SerialExecutor.execute(SerialExecutor.java:69)
at io.fabric8.kubernetes.client.dsl.internal.ExecWebSocketListener.onClose(ExecWebSocketListener.java:373)
at io.fabric8.kubernetes.client.vertx.VertxWebSocket.lambda$init$2(VertxWebSocket.java:55)
at io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:270)
at io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:252)
at io.vertx.core.http.impl.WebSocketImplBase.receiveFrame(WebSocketImplBase.java:560)
at io.vertx.core.streams.impl.InboundBuffer.handleEvent(InboundBuffer.java:279)
at io.vertx.core.streams.impl.InboundBuffer.write(InboundBuffer.java:157)
at io.vertx.core.http.impl.WebSocketImplBase.handleFrame(WebSocketImplBase.java:492)
at io.vertx.core.impl.ContextImpl.execute(ContextImpl.java:327)
at io.vertx.core.impl.ContextImpl.execute(ContextImpl.java:307)
at io.vertx.core.http.impl.Http1xConnectionBase.handleWsFrame(Http1xConnectionBase.java:66)
at io.vertx.core.http.impl.Http1xClientConnection.handleMessage(Http1xClientConnection.java:778)
at io.vertx.core.net.impl.ConnectionBase.read(ConnectionBase.java:159)
at io.vertx.core.net.impl.VertxHandler.channelRead(VertxHandler.java:153)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1503)
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1366)
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1415)
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:530)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:469)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1357)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:868)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:840)

@shawkins
Copy link
Contributor

Same issue reported in major version 6.13.3 fixed in 6.13.4 is also occurring 7.0.1

This is happening in a different place, but for the same reason. We can simply catch the rejected execution exception in the onClose method. Until it's addressed, these logs can safely be ignored.

shawkins added a commit to shawkins/kubernetes-client that referenced this issue Jan 3, 2025
@shawkins
fix: catching rejected execution
a613c0a 
closes: fabric8io#6747

Signed-off-by: Steve Hawkins <[email protected]>
@shawkins shawkins mentioned this issue Jan 3, 2025
Merged
11 tasks
@manusa manusa added this to the 7.1.0 milestone Jan 7, 2025 — with automated-tasks
manusa pushed a commit that referenced this issue Jan 7, 2025
@shawkins
fix: catching rejected execution
ebeec26 
closes: #6747

Signed-off-by: Steve Hawkins <[email protected]>
@wayneguow wayneguow mentioned this issue Feb 13, 2025
Closed
dongjoon-hyun pushed a commit to apache/spark that referenced this issue Feb 13, 2025
@wayneguow @dongjoon-hyun
[SPARK-51195][BUILD][K8S] Upgrade kubernetes-client to 7.1.0
ae2d2e8 
### What changes were proposed in this pull request?

This PR aims to upgrade `kubernetes-client` from 7.0.1 to 7.1.0.

### Why are the changes needed?

Keep the dependencies to the latest, and there are some bug fixes in version 7.1.0:
- Fix fabric8io/kubernetes-client#6725: (crd-generator) CRD generator missing type for GenericKubernetesResource
- Fix fabric8io/kubernetes-client#6747: Preventing websocket error logs when the client is closed
- Fix fabric8io/kubernetes-client#6781: Allowing ipv6 entries to work in NO_PROXY
- Fix fabric8io/kubernetes-client#6792: VertxHttpClient uses exclusive Vert.x instance by default

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

Pass GA.

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes #49925 from wayneguow/kubernetes-client.

Authored-by: Wei Guo <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
dongjoon-hyun pushed a commit to apache/spark that referenced this issue Feb 14, 2025
@wayneguow @dongjoon-hyun
[SPARK-51195][BUILD][K8S] Upgrade kubernetes-client to 7.1.0
dcb3b44 
### What changes were proposed in this pull request?

This PR aims to upgrade `kubernetes-client` from 7.0.1 to 7.1.0.

### Why are the changes needed?

Keep the dependencies to the latest, and there are some bug fixes in version 7.1.0:
- Fix fabric8io/kubernetes-client#6725: (crd-generator) CRD generator missing type for GenericKubernetesResource
- Fix fabric8io/kubernetes-client#6747: Preventing websocket error logs when the client is closed
- Fix fabric8io/kubernetes-client#6781: Allowing ipv6 entries to work in NO_PROXY
- Fix fabric8io/kubernetes-client#6792: VertxHttpClient uses exclusive Vert.x instance by default

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

Pass GA.

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes #49925 from wayneguow/kubernetes-client.

Authored-by: Wei Guo <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
(cherry picked from commit ae2d2e8)
Signed-off-by: Dongjoon Hyun <[email protected]>
kazemaksOG pushed a commit to kazemaksOG/spark-custom-scheduler that referenced this issue Mar 27, 2025
@wayneguow @kazemaksOG
[SPARK-51195][BUILD][K8S] Upgrade kubernetes-client to 7.1.0
286fc9d 
### What changes were proposed in this pull request?

This PR aims to upgrade `kubernetes-client` from 7.0.1 to 7.1.0.

### Why are the changes needed?

Keep the dependencies to the latest, and there are some bug fixes in version 7.1.0:
- Fix fabric8io/kubernetes-client#6725: (crd-generator) CRD generator missing type for GenericKubernetesResource
- Fix fabric8io/kubernetes-client#6747: Preventing websocket error logs when the client is closed
- Fix fabric8io/kubernetes-client#6781: Allowing ipv6 entries to work in NO_PROXY
- Fix fabric8io/kubernetes-client#6792: VertxHttpClient uses exclusive Vert.x instance by default

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

Pass GA.

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes apache#49925 from wayneguow/kubernetes-client.

Authored-by: Wei Guo <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@shawkins shawkins

Labels
None yet
Projects
None yet
Milestone
7.1.0
3 participants
@manusa @shawkins @vicozohili