KAFKA-16758: Extend Consumer#close with an option to leave the group or not #17614

frankvicky · 2024-10-28T03:42:08Z

JIRA: KAFKA-16758
This PR is aim to deliver
KIP-1092,
please refer to KIP-1092 and KAFKA-16758 for further details.

Reviewers: Anna Sophie Blee-Goldman [email protected],
Chia-Ping Tsai [email protected], Kirk True [email protected]

...nts/src/main/java/org/apache/kafka/clients/consumer/internals/AbstractMembershipManager.java

frankvicky · 2024-10-28T04:08:13Z

...nts/src/main/java/org/apache/kafka/clients/consumer/internals/AbstractMembershipManager.java

+     *
+     * @see GroupMembershipOperation
+     */
+    protected GroupMembershipOperation leaveGroupOperation = GroupMembershipOperation.DEFAULT;


Due to the architecture of the async consumer, it's difficult to pass GroupMembershipOperation directly to AbstractMembershipManager, so I added a property to hold the behavior for leaving the group, which remains unchanged except when the consumer is closing.

...nts/src/test/java/org/apache/kafka/clients/consumer/internals/ConsumerNetworkThreadTest.java

frankvicky · 2024-11-04T14:51:17Z

Hello @ableegoldman @mjsax @chia7712
Could you please take a look at the PR when you have some time ?
Many thanks 😺

ableegoldman

Made a quick pass though it would be good if someone more familiar with the consumer internals can take a look as well.

Also, I take it you plan to fix the KafkaStreams#close behavior and remove the internal.leave.group.on.close flag in a separate followup PR?

ableegoldman · 2024-11-09T04:50:59Z

clients/src/main/java/org/apache/kafka/clients/consumer/GroupMembershipOperation.java

+ * Enum to specify the group membership operation upon leaving group.
+ * {@code LEAVE_GROUP} means the consumer will leave the group.
+ * {@code REMAIN_IN_GROUP} means the consumer will remain in the group.
+ * {@code DEFAULT} applies the default behavior, which may depend on whether the consumer is static or dynamic.


Should we clarify what the actual behavior is for static vs dynamic?

That's a good point. I'll include some additional description for it. 😄

ableegoldman · 2024-11-09T04:57:48Z

clients/src/main/java/org/apache/kafka/clients/consumer/internals/AbstractCoordinator.java

+                // If membershipOperation is DEFAULT, leave group based on rebalanceConfig.leaveGroupOnClose.
+                // Otherwise, leave group only if membershipOperation is LEAVE_GROUP.
+                if (GroupMembershipOperation.REMAIN_IN_GROUP != membershipOperation &&
+                    (GroupMembershipOperation.LEAVE_GROUP == membershipOperation || rebalanceConfig.leaveGroupOnClose)) {


the "leaveGroupOnClose" rebalance config is an internal backdoor that we put in for Streams to get around the lack of this API -- so once we update the corresponding KafkaStreams# close method we should remember to remove this config and clean up the code. The secret "internal" config was always a bit hacky so it's nice to finally get rid of it

Also, it's a bit weird to be applying this check in two places. I think we should remove this check entirely and only have the one in #maybeLeaveGroup

(Frankly it was kind of weird to put the leaveGroupOnClose check here to begin with, imo it should have been part of the #maybeLeaveGroup check to begin with. So I'd just remove the entire if condition and move the leaveGroupOnClose config check to the #maybeLeaveGroup check until we can remove it entirely

Am I correct in thinking that if we decide to move the if condition into maybeLeaveGroup,
we should also move onLeavePrepare into maybeLeaveGroup?

If that’s the case, it might be tricky.
Currently, I assume that onLeavePrepare and maybeLeaveGroup should be invoked consecutively,
since they share the same if condition. However, at the moment, there are many places
in both test code and production code that only invoke maybeLeaveGroup.

If we proceed with this refactor, we will need to ensure that we do not break the current behavior.
Here’s a rough idea:

public synchronized RequestFuture<Void> maybeLeaveGroup(GroupMembershipOperation membershipOperation, String leaveReason) { // If membershipOperation is REMAIN_IN_GROUP, never send leave group request. if (GroupMembershipOperation.REMAIN_IN_GROUP == membershipOperation) { return null; } // If membershipOperation is DEFAULT, leave group based on rebalanceConfig.leaveGroupOnClose. // Otherwise, leave group only if membershipOperation is LEAVE_GROUP. if (GroupMembershipOperation.LEAVE_GROUP != membershipOperation && !rebalanceConfig.leaveGroupOnClose) { return null; } onLeavePrepare(); return leaveGroup(membershipOperation, leaveReason); } public RequestFuture<Void> leaveGroup(GroupMembershipOperation membershipOperation, String leaveReason) { RequestFuture<Void> future = null; if (shouldSendLeaveGroupRequest(membershipOperation)) { log.info("Member {} sending LeaveGroup request to coordinator {} due to {}", generation.memberId, coordinator, leaveReason); LeaveGroupRequest.Builder request = new LeaveGroupRequest.Builder( rebalanceConfig.groupId, List.of(new MemberIdentity().setMemberId(generation.memberId).setReason(JoinGroupRequest.maybeTruncateReason(leaveReason))) ); future = client.send(coordinator, request).compose(new LeaveGroupResponseHandler(generation)); client.pollNoWakeup(); } resetGenerationOnLeaveGroup(); return future; }

We can move the if condition and onLeavePrepare into maybeLeaveGroup, while extracting
the original logic of maybeLeaveGroup into a new public helper method.
Code that previously invoked maybeLeaveGroup directly will instead call the new helper method.
WDYT ?

I agree with @ableegoldman that we should check the new option in maybeLeaveGroup. Additionally, since we haven't discussed it in the KIP, should we skip the callback if the state is REMAIN_IN_GROUP? I prefer not to modify the callback to minimize changes

I will initiate a discussion on our KIP thread.

replied on the KIP discussion thread. After some thought it's actually a bit more complicated than I initially imagined -- so I'd be interested to hear all your thoughts

clients/src/main/java/org/apache/kafka/clients/consumer/internals/AbstractCoordinator.java

clients/src/main/java/org/apache/kafka/clients/consumer/internals/ClassicKafkaConsumer.java

clients/src/main/java/org/apache/kafka/clients/consumer/internals/CloseOptionInternal.java

frankvicky · 2024-11-16T08:34:01Z

Hi @lianetm,
Could you please take a look?
Since this PR includes some changes to the consumer, it would be nice if you could share your thoughts.
Many thanks! 🙇🏼

TaiJuWu · 2024-11-27T00:33:59Z

clients/src/main/java/org/apache/kafka/clients/consumer/Consumer.java

+         * Fluent method to set the group membership operation upon shutdown.
+         *
+         * @param operation the group membership operation to apply. Must be one of {@code LEAVE_GROUP}, {@code REMAIN_IN_GROUP},
+     *                 or {@code DEFAULT}.


nit: align start

kirktrue

Thanks for the PR @frankvicky.

I didn't make it as far as the tests, but I left a few comments. Thanks!

...nts/src/main/java/org/apache/kafka/clients/consumer/internals/AbstractMembershipManager.java

clients/src/main/java/org/apache/kafka/clients/consumer/internals/CloseOptionInternal.java

kirktrue · 2024-12-14T01:13:27Z

...c/main/java/org/apache/kafka/clients/consumer/internals/AbstractHeartbeatRequestManager.java

+        AbstractMembershipManager<R> membershipManager = membershipManager();
+        GroupMembershipOperation leaveGroupOperation = membershipManager.leaveGroupOperation();
+
+        if (membershipManager.isLeavingGroup() &&
+            // Default operation: both static and dynamic consumers will send a leave heartbeat
+            (GroupMembershipOperation.DEFAULT == leaveGroupOperation ||
+                // Leave operation: both static and dynamic consumers will send a leave heartbeat
+                GroupMembershipOperation.LEAVE_GROUP == leaveGroupOperation ||
+                // Remain in group: only static consumers will send a leave heartbeat, while dynamic members will not
+                membershipManager.groupInstanceId().isPresent())
+        ) {


Could we let this logic live inside the MembershipManager.isLeavingGroup() implementation rather than exposing it here? The MembershipManager already manages the group instance ID and GroupMembershipOperation values internally. Then we wouldn't have to add the other AbstractMembershipManager changes, right?

Reminder on this question.

chia7712

@frankvicky thanks for this patch. a couple of comments are left. PTAL

chia7712 · 2024-12-16T12:58:49Z

clients/src/main/java/org/apache/kafka/clients/consumer/Consumer.java

+     */
+    void close(final CloseOption option);
+
+    class CloseOption {


Perhaps we can move the close method out of the Consumer class. Currently, users need to call consumer.close(Consumer.CloseOption.timeout(Duration.xxx)), which is somewhat verbose. Simplifying this could enhance the user experience. :(

chia7712 · 2024-12-16T12:59:15Z

clients/src/main/java/org/apache/kafka/clients/consumer/Consumer.java

+         *                 or {@code DEFAULT}.
+         * @return a new {@code CloseOption} instance with the specified group membership operation.
+         */
+        public static CloseOption groupMembershipOperation(final GroupMembershipOperation operation) {


Could you please file a follow-up to increase the test coverage?

https://issues.apache.org/jira/browse/KAFKA-18267

chia7712 · 2024-12-16T12:59:43Z

clients/src/main/java/org/apache/kafka/clients/consumer/Consumer.java

+         * @return this {@code CloseOption} instance.
+         */
+        public CloseOption withGroupMembershipOperation(final GroupMembershipOperation operation) {
+            this.operation = operation;


please add null check

chia7712 · 2024-12-16T12:59:50Z

clients/src/main/java/org/apache/kafka/clients/consumer/Consumer.java

+         */
+        public static CloseOption groupMembershipOperation(final GroupMembershipOperation operation) {
+            CloseOption option = new CloseOption();
+            option.operation = operation;


please add null check

clients/src/main/java/org/apache/kafka/clients/consumer/internals/AsyncKafkaConsumer.java

clients/src/main/java/org/apache/kafka/clients/consumer/internals/ClassicKafkaConsumer.java

chia7712 · 2024-12-16T13:06:47Z

...nts/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerMembershipManager.java

-        return groupInstanceId;
+    @Override
+    public void leaveGroupOperationOnClose(GroupMembershipOperation operation) {
+        if (GroupMembershipOperation.DEFAULT.equals(operation)) {


why do we need those check? we eventually need to handle the DEFAULT for different cases, right?

chia7712 · 2024-12-16T13:26:37Z

clients/src/main/java/org/apache/kafka/clients/consumer/internals/AbstractCoordinator.java

+                // If membershipOperation is DEFAULT, leave group based on rebalanceConfig.leaveGroupOnClose.
+                // Otherwise, leave group only if membershipOperation is LEAVE_GROUP.
+                if (GroupMembershipOperation.REMAIN_IN_GROUP != membershipOperation &&
+                    (GroupMembershipOperation.LEAVE_GROUP == membershipOperation || rebalanceConfig.leaveGroupOnClose)) {


I agree with @ableegoldman that we should check the new option in maybeLeaveGroup. Additionally, since we haven't discussed it in the KIP, should we skip the callback if the state is REMAIN_IN_GROUP? I prefer not to modify the callback to minimize changes

kirktrue

Thanks @frankvicky. Left some more comments.

kirktrue · 2025-01-06T23:15:45Z

...c/main/java/org/apache/kafka/clients/consumer/internals/AbstractHeartbeatRequestManager.java

+        AbstractMembershipManager<R> membershipManager = membershipManager();
+        GroupMembershipOperation leaveGroupOperation = membershipManager.leaveGroupOperation();
+
+        if (membershipManager.isLeavingGroup() &&
+            // Default operation: both static and dynamic consumers will send a leave heartbeat
+            (GroupMembershipOperation.DEFAULT == leaveGroupOperation ||
+                // Leave operation: both static and dynamic consumers will send a leave heartbeat
+                GroupMembershipOperation.LEAVE_GROUP == leaveGroupOperation ||
+                // Remain in group: only static consumers will send a leave heartbeat, while dynamic members will not
+                membershipManager.groupInstanceId().isPresent())
+        ) {


Reminder on this question.

...nts/src/main/java/org/apache/kafka/clients/consumer/internals/AbstractMembershipManager.java

clients/src/main/java/org/apache/kafka/clients/consumer/internals/AsyncKafkaConsumer.java

ableegoldman

Almost there!

ableegoldman · 2025-04-03T01:51:50Z

...c/main/java/org/apache/kafka/clients/consumer/internals/ConsumerHeartbeatRequestManager.java

+        // If the consumer has dynamic membership,
+        // we should skip the leaving heartbeat when leaveGroupOperation is REMAIN_IN_GROUP
+        boolean skipHeartbeatForDynamicMemberRemainInGroup = membershipManager.groupInstanceId().isEmpty()
+            && REMAIN_IN_GROUP == membershipManager.leaveGroupOperation();


very minor nit: the double negative with "!skipX" is a bit confusing, instead of "skipHeartbeat..." can we invert the variable to "shouldHeartbeat"?

ableegoldman · 2025-04-03T02:00:10Z

...c/main/java/org/apache/kafka/clients/consumer/internals/AbstractHeartbeatRequestManager.java

@@ -182,7 +181,7 @@ public NetworkClientDelegate.PollResult poll(long currentTimeMs) {
        }

        // Case 1: The member is leaving
-        boolean heartbeatNow = membershipManager().state() == MemberState.LEAVING ||
+        boolean heartbeatNow = isLeavingGroup() ||


why add the #isLeavingGroup method to the AbstractHeartbeatRequestManager instead of just calling membershipManager#isLeavingGroup here?

Basically I'm wondering why both the HeartbeatRequestManager and MembershipManager classes need to have this #isLeavingGroup method

membershipManager#isLeavingGroup considers LEAVING and PREPARE_LEAVING to be leaving the group.
But in this flag, we only consider LEAVING since the flag determines whether the heartbeat should be sent immediately or not. PREPARE_LEAVING is the state that waits for callbacks to be executed.

The key point is that it's confusing that HeartbeatRequestManager and MembershipManager have a method with the same name. Clearly, HeartbeatRequestManager cares about the timing of a leave request, not the member state.
I will figure out a more appropriate name.

ableegoldman · 2025-04-05T02:03:05Z

...c/main/java/org/apache/kafka/clients/consumer/internals/ConsumerHeartbeatRequestManager.java

+        // Determine if we should send a leaving heartbeat:
+        // - For static membership (when groupInstanceId is present): Always send the leaving heartbeat
+        // - For dynamic membership: Send the leaving heartbeat only when leaveGroupOperation is not REMAIN_IN_GROUP
+        boolean shouldHeartbeat = membershipManager.groupInstanceId().isPresent()


IIUC the only difference in this #pollOnClose implementation from the original one in the abstract class is the addition of this shouldHeartbeat condition to the if membershipManager().isLeavingGroup() check.

However it seems like the extra condition is built entirely of calls to membershipManager methods so I'm wondering if it doesn't make more sense to just incorporate the shouldHeartbeat condition into the #isLeavingGroup method of the ConsumerMembershipManager class? That way we don't need to make #pollOnClose abstract and can cut down on some redundant code

Also from a quick look at the ConsumerMembershipManager#isLeavingGroup implementation, it already includes checks on both of these conditions (groupInstanceId.isPresentandREMAIN_IN_GROUP == membershipManager.leaveGroupOperation()). imo it would be cleaner and easier to read if we just encapsulated all the logic around these conditions inside the #isLeavingGroup` method

Yes, this could significantly reduce redundancy!
Previously, I thought it was more clear that the async consumer and share consumer have their implementation, but it does not provide too much value now.

ableegoldman · 2025-04-05T02:19:18Z

...nts/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerMembershipManager.java

-                ConsumerGroupHeartbeatRequest.LEAVE_GROUP_STATIC_MEMBER_EPOCH :
-                ConsumerGroupHeartbeatRequest.LEAVE_GROUP_MEMBER_EPOCH;
+        boolean isStaticMember = groupInstanceId.isPresent();
+        if (REMAIN_IN_GROUP == leaveGroupOperation && isStaticMember) {


this condition is basically redundant, right? if isStaticMember is true then we'll return LEAVE_GROUP_STATIC_MEMBER_EPOCH anyways via the final check

Yes, I should clean this method up after #17614 (comment)
Now, the matrix of membership and epoch is :

REMAIN LEAVE DEFAULT

dynamic Don't send -1 -1

static -2 -1 -2

Given that, we could remove if (REMAIN_IN_GROUP == leaveGroupOperation && isStaticMember) condition.

ableegoldman · 2025-04-05T02:33:37Z

...nts/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerMembershipManager.java

+        if (LEAVE_GROUP == leaveGroupOperation) {
+            return ConsumerGroupHeartbeatRequest.LEAVE_GROUP_MEMBER_EPOCH;


I think I understand why this condition is needed but it would be good to leave a comment explaining this change because it's pretty obscure. IIUC basically the problem is that there's no way to permanently leave the group with a static consumer right now, and the only method of removing static members is via an admin API. So we essentially have to "trick" the GroupMetadataManager into fencing this member to kick it out of the group, because if we use the LEAVE_GROUP_STATIC_MEMBER_EPOCH then it thinks it's just being "temporarily" removed and this won't actually result in the consumer leaving the group. So we use LEAVE_GROUP_MEMBER_EPOCH because anything that isn't the LEAVE_GROUP_STATIC_MEMBER_EPOCH triggers the GroupMetadataManager to fence the consumer in its #consumerGroupLeave method.

First off does that summary sound right? It's also pretty hacky, and while I think it's fine to proceed with this for now, I'm wondering if we shouldn't just try to implement proper LeaveGroup mechanics for static membership. I know this would require some server-side changes but maybe this can be a followup KIP? Again, no need to block this PR on that, just putting it out there. Feel free to pick that up yourself or file a ticket if that makes sense to you too

Your summary is entirely correct. I will add a comment in the next commit.

I know this would require some server-side changes but maybe this can be a followup KIP? Again, no need to block this PR on that, just putting it out there. Feel free to pick that up yourself or file a ticket if that makes sense to you too.

I will file a ticket to trace this one and see if we can gather some feedback.

https://issues.apache.org/jira/browse/KAFKA-19094

ableegoldman

Awesome, I think we're finally good to go with this! Thanks for picking this up and for being patient, this turned out to be more complex than it seemed and there were a lot of details to work out 🙂

Will merge to trunk

ableegoldman · 2025-04-06T05:07:44Z

Merged to trunk (🥳 ) -- do you want to start working on the followup PR now to utilize this new API for KafkaStreams#close? This will also let us clean up the internal.leave.group.on.close config that streams has been using as a backdoor. Now that we have the REMAIN_IN_GROUP option we can use that as the default for Streams and get rid of the internal config entirely

frankvicky · 2025-04-06T05:58:23Z

Hi @ableegoldman
A huge thanks for your patience and review! 🙇🏼

do you want to start working on the followup PR now to utilize this new API for KafkaStreams#close?

Yes, I will start to tackle it soon 🚀
I will ping you once the patch is ready.

Thanks again 😸

mjsax · 2025-04-24T02:40:32Z

Great to see this merged. I took the liberty to do small follow up PR to improve the JavaDocs: #19546

github-actions bot added consumer connect storage Pull requests that target the storage module clients labels Oct 28, 2024

frankvicky commented Oct 28, 2024

View reviewed changes

...nts/src/main/java/org/apache/kafka/clients/consumer/internals/AbstractMembershipManager.java Outdated Show resolved Hide resolved

frankvicky commented Oct 28, 2024

View reviewed changes

...nts/src/test/java/org/apache/kafka/clients/consumer/internals/ConsumerNetworkThreadTest.java Outdated Show resolved Hide resolved

frankvicky added the ci-approved label Oct 28, 2024

frankvicky force-pushed the KAFKA-16758 branch from e4e6468 to a196516 Compare November 2, 2024 01:12

frankvicky marked this pull request as ready for review November 2, 2024 01:12

frankvicky force-pushed the KAFKA-16758 branch from a196516 to 73205b4 Compare November 2, 2024 02:14

frankvicky requested review from ableegoldman, chia7712 and mjsax November 4, 2024 14:49

ableegoldman reviewed Nov 15, 2024

View reviewed changes

frankvicky force-pushed the KAFKA-16758 branch 2 times, most recently from 53ffc57 to b457537 Compare November 16, 2024 14:28

TaiJuWu reviewed Nov 27, 2024

View reviewed changes

frankvicky force-pushed the KAFKA-16758 branch from b457537 to 653ce16 Compare December 6, 2024 13:14

github-actions bot added the tiered-storage Related to the Tiered Storage feature label Dec 6, 2024

frankvicky force-pushed the KAFKA-16758 branch from 653ce16 to bdb5113 Compare December 6, 2024 13:15

kirktrue reviewed Dec 14, 2024

View reviewed changes

kirktrue added KIP-848 The Next Generation of the Consumer Rebalance Protocol ctr Consumer Threading Refactor (KIP-848) labels Dec 14, 2024

frankvicky force-pushed the KAFKA-16758 branch from bdb5113 to ab69d48 Compare December 16, 2024 11:10

chia7712 reviewed Dec 16, 2024

View reviewed changes

frankvicky force-pushed the KAFKA-16758 branch from ab69d48 to 8030579 Compare December 16, 2024 16:09

kirktrue reviewed Jan 6, 2025

View reviewed changes

frankvicky added 14 commits March 15, 2025 12:54

Merge branch 'trunk' into KAFKA-16758

3e5e288

Merge branch 'trunk' into KAFKA-16758

7513446

Merge branch 'trunk' into KAFKA-16758

3d4ed4a

Merge branch 'trunk' into KAFKA-16758

9d45444

Merge branch 'trunk' into KAFKA-16758

1e18a34

Merge branch 'trunk' into KAFKA-16758

204cb83

Merge branch 'trunk' into KAFKA-16758

fb4055b

Merge branch 'trunk' into KAFKA-16758

e5c9fa5

Merge branch 'trunk' into KAFKA-16758

ab89078

Merge branch 'trunk' into KAFKA-16758

61a6346

Merge branch 'trunk' into KAFKA-16758

fbc878a

Merge branch 'trunk' into KAFKA-16758

6143d52

Merge branch 'trunk' into KAFKA-16758

17d3ee2

Merge branch 'trunk' into KAFKA-16758

3bf1e26

ableegoldman reviewed Apr 3, 2025

View reviewed changes

frankvicky added 3 commits April 4, 2025 23:39

Merge branch 'trunk' into KAFKA-16758

11fb74f

Merge branch 'trunk' into KAFKA-16758

076ea2a

Address comments

e946499

ableegoldman reviewed Apr 5, 2025

View reviewed changes

frankvicky added 3 commits April 5, 2025 19:45

Address comments

8ee016d

change scope to private

7fc4237

Fix failed tests

f37d223

ableegoldman approved these changes Apr 6, 2025

View reviewed changes

ableegoldman merged commit 74acbd2 into apache:trunk Apr 6, 2025
22 of 23 checks passed

frankvicky mentioned this pull request Apr 7, 2025

KAFKA-18185: remove internal.leave.group.on.close config #19400

Open

		if (LEAVE_GROUP == leaveGroupOperation) {
		return ConsumerGroupHeartbeatRequest.LEAVE_GROUP_MEMBER_EPOCH;

KAFKA-16758: Extend Consumer#close with an option to leave the group or not #17614

KAFKA-16758: Extend Consumer#close with an option to leave the group or not #17614

Conversation

frankvicky commented Oct 28, 2024 • edited by github-actions bot Loading

Choose a reason for hiding this comment

frankvicky commented Nov 4, 2024

ableegoldman left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

frankvicky commented Nov 16, 2024

Choose a reason for hiding this comment

kirktrue left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chia7712 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kirktrue left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ableegoldman left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

frankvicky Apr 5, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ableegoldman left a comment

Choose a reason for hiding this comment

ableegoldman commented Apr 6, 2025

frankvicky commented Apr 6, 2025 • edited Loading

mjsax commented Apr 24, 2025

frankvicky commented Oct 28, 2024 •

edited by github-actions bot

Loading

frankvicky Apr 5, 2025 •

edited

Loading

frankvicky commented Apr 6, 2025 •

edited

Loading