Warn about duplicated events received from GitHub via Admin Monitor #388

gbhat618 · 2025-02-03T12:54:10Z

Problem
If duplicated GitHub events get delivered to Jenkins controller, Jenkins need to act on events both times (they are not automatically de-duplicated). These events may be a branch scan, build trigger etc.

further discussion of problem:

A branch scan event running twice may not be a problem, however build triggering twice could be a problem. Even the builds doesn't get in triggered twice due to quietPeriod: 5s (unless the quietPeriod was set to 0, or duplicates come after quietPeriod: 5s, which case two builds do get triggered).

Note: usually GH by itself won't automatically redeliver an event if failed, GH simply marks the event as failed (an event is marked failed if GH didn't receive a 200 OK response within 10s; reference)

The reason for such duplicated events is,

mostly, a wrong configuration in GitHub side, such as registering two webhooks (maybe at multiple places such as org, and repo levels) which are targeting the same controller.
if a custom setup is done to redeliver failed events if the redeliver is going wrong (this should be rare to never)

Although it is typically rare for this duplication issue to happen, if it happens, it could require quite some time to figure out what is going on. And in some case it may simply go unnoticed.

Logs showing build can trigger twice in case of `quietPeriod: 0`

Note: in the below logs, when the two events came

at exactly the same second, Jenkins didn't execute 2 builds, rather both the events corresponded to a single build; except in the build log we saw Started by GitHub push by gbhat618 twice (ref); however rest of logs showed build steps were executed only once; I verified by this by having a single executor.
within 1 second apart, Jenkins executed two builds one after other.

2025-02-06 03:20:19.393+0000 [id=87]	INFO	o.j.p.g.w.s.DefaultPushGHEventSubscriber#onEvent: Received PushEvent for https://github.com/gbhat618/dummy from 140.82.115.146:35792 → 140.82.115.146 ⇒ https://localhost:8080/github-webhook/
2025-02-06 03:20:19.393+0000 [id=88]	INFO	o.j.p.g.w.s.DefaultPushGHEventSubscriber#onEvent: Received PushEvent for https://github.com/gbhat618/dummy from 140.82.115.168:30022 → 140.82.115.168 ⇒ https://localhost:8080/github-webhook/
2025-02-06 03:20:19.432+0000 [id=88]	INFO	o.j.p.g.w.s.DefaultPushGHEventSubscriber$1#run: Poked p1
2025-02-06 03:20:19.432+0000 [id=87]	INFO	o.j.p.g.w.s.DefaultPushGHEventSubscriber$1#run: Poked p1
2025-02-06 03:20:20.133+0000 [id=90]	INFO	c.c.jenkins.GitHubPushTrigger$1#run: SCM changes detected in p1. Triggering #6
2025-02-06 03:20:20.133+0000 [id=91]	INFO	c.c.jenkins.GitHubPushTrigger$1#run: SCM changes detected in p1. Triggering #6
2025-02-06 03:20:35.660+0000 [id=123]	INFO	o.j.p.g.ApiRateLimitChecker$RateLimitCheckerAdapter#checkRateLimit: LocalChecker for rate limit was not set for this thread. Configured using system settings with API URL 'https://api.github.com'.
2025-02-06 03:21:12.640+0000 [id=19]	INFO	o.j.p.g.w.s.DefaultPushGHEventSubscriber#onEvent: Received PushEvent for https://github.com/gbhat618/dummy from 140.82.115.172:50884 → 140.82.115.172 ⇒ https://localhost:8080/github-webhook/
2025-02-06 03:21:12.644+0000 [id=174]	INFO	o.j.p.g.w.s.DefaultPushGHEventSubscriber#onEvent: Received PushEvent for https://github.com/gbhat618/dummy from 140.82.115.82:36500 → 140.82.115.82 ⇒ https://localhost:8080/github-webhook/
2025-02-06 03:21:12.649+0000 [id=174]	INFO	o.j.p.g.w.s.DefaultPushGHEventSubscriber$1#run: Poked p1
2025-02-06 03:21:12.649+0000 [id=19]	INFO	o.j.p.g.w.s.DefaultPushGHEventSubscriber$1#run: Poked p1
2025-02-06 03:21:13.301+0000 [id=185]	INFO	c.c.jenkins.GitHubPushTrigger$1#run: SCM changes detected in p1. Triggering #7
2025-02-06 03:21:13.301+0000 [id=186]	INFO	c.c.jenkins.GitHubPushTrigger$1#run: SCM changes detected in p1. Triggering #7
2025-02-06 03:21:30.552+0000 [id=122]	INFO	o.j.p.g.ApiRateLimitChecker$RateLimitCheckerAdapter#checkRateLimit: LocalChecker for rate limit was not set for this thread. Configured using system settings with API URL 'https://api.github.com'.
2025-02-06 03:21:47.226+0000 [id=19]	INFO	o.j.p.g.w.s.DefaultPushGHEventSubscriber#onEvent: Received PushEvent for https://github.com/gbhat618/dummy from 140.82.115.11:19518 → 140.82.115.11 ⇒ https://localhost:8080/github-webhook/
2025-02-06 03:21:47.226+0000 [id=171]	INFO	o.j.p.g.w.s.DefaultPushGHEventSubscriber#onEvent: Received PushEvent for https://github.com/gbhat618/dummy from 140.82.115.73:36722 → 140.82.115.73 ⇒ https://localhost:8080/github-webhook/
2025-02-06 03:21:47.229+0000 [id=171]	INFO	o.j.p.g.w.s.DefaultPushGHEventSubscriber$1#run: Poked p1
2025-02-06 03:21:47.229+0000 [id=19]	INFO	o.j.p.g.w.s.DefaultPushGHEventSubscriber$1#run: Poked p1
2025-02-06 03:21:47.874+0000 [id=242]	INFO	c.c.jenkins.GitHubPushTrigger$1#run: SCM changes detected in p1. Triggering #8
2025-02-06 03:21:47.889+0000 [id=241]	INFO	c.c.jenkins.GitHubPushTrigger$1#run: SCM changes detected in p1. Triggering #9

Proposed solution

An admin monitor to show duplicate events are being received - admin monitor goes away by itself after 24 hours if no more duplicates.

further discussion of solution:

Duplication of event is identified by a header X-GitHub-Delivery (reference, also linked in javadoc) which is a unique identifier for an event in GitHub.

An admin monitor is shown to warn about duplicated events being received. If the duplicated events are to occur, they typically occur in a few seconds window itself, we track an event for duplicate checking for about 10 minutes - so it should be enough to catch the issue.

Event tracking cache:
Caffeine is used for caching via caffeine-api-plugin. The cache max size is attempted to be kept at ~1MB by setting max size, and ttl (further notes in javadoc).

The admin can find the last received duplicate payload via the Click Here button shown (see the screenshots in Testing section) -- it opens the payload JSON in the new browser tab next to current tab. (browsers like firefox and chrome are by default acting as pretty json viewer - due to content-type application/json)

I have considered alternative ways of presenting the event payload as well,

Logging the event payload
- Typically users tend to take a look at the problem if it is shown on the UI.
- In the logs, events get missed. Besides we need to also consider while coding, payload is very huge for log, making sure to log the event only once (thread synchronization), how often to log the event. So logging doesn't seem like a good user experience for the admin.
Automatically downloading a JSON file upon Click Here: most of the admins can take a decision directly from seeing the payload in the browser (downloading to laptop - is another problem for the admin to cleanup the file later)
- The payload is not that huge as in to not be visible in the UI (or crashing the browser etc.). It is within the limit of ~200 lines in pretty format.
- There are not multiple files (such as how the jenkins support bundle would have)
- Browsers like, firefox and chrome can automatically pretty format the JSON, so it is also acting as a JSON viewer in this case.

Testing done

Automated tests are submitted
- mimicking the GH event sending via triggering webhook endpoint; and validating admin monitor appears in case of duplication
- unit tests to assert logic of duplicate events computation and cleanup validation.
Manual testing
- Registering two webhooks on a github repo, and trigger commits on a branch; jenkins received 2 events. Admin monitor showed. (also two builds do get triggered for setting quietPeriod: 0).
- Monitor goes away after 24h.

Screenshots

Desc	Screenshot
Admin monitor appears when duplicate events are discovered
Click here button opens in the new browser tab with the event JSON
If somehow admin goes to the above json payload page - but there are no duplicates

Submitter checklist

Make sure you are opening from a topic/feature/bugfix branch (right side) and not your main branch!
Ensure that the pull request title represents the desired changelog entry
Please describe what you did
Link to relevant issues in GitHub or Jira
Link to relevant pull requests, esp. upstream and downstream changes
Ensure you have provided tests - that demonstrates feature works or fixes the issue

gbhat618 · 2025-02-03T12:54:33Z

👋 @Vlatombe , @jglick

Vlatombe

LGTM. Made some suggestions.

src/main/java/org/jenkinsci/plugins/github/extension/GHSubscriberEvent.java

src/main/java/org/jenkinsci/plugins/github/webhook/subscriber/DuplicateEventsSubscriber.java

…DuplicateEventsSubscriber.java Co-authored-by: Vincent Latombe <[email protected]>

…admin monitor (checking if this is the best, if yes then will write a test for it)

gbhat618 · 2025-02-07T04:12:48Z

I am fixing how the last duplicate event is presented to the user from, enable FINEST logger (current implementation) to click here to download

Edit: I made the payload open in a new browser tab since browsers like Firefox and Chrome format JSON nicely. With ~200 lines, it's easy to view without issues. The admin can still download it if needed or simply review it in the browser and proceed.

src/main/java/org/jenkinsci/plugins/github/admin/GitHubDuplicateEventsMonitor.java

gbhat618 · 2025-02-07T16:39:30Z

I think this PR is now ready 😄 I feel like covered all the features, and automated tests and manual tests.
Please help review, ty

gbhat618 · 2025-02-10T13:27:25Z

(the CI failure is infra issue in windows (linux already passed), I am currently thinking to push new commits are a PR review, that way the CI will retrigger)

src/main/java/org/jenkinsci/plugins/github/extension/GHSubscriberEvent.java

src/main/java/org/jenkinsci/plugins/github/webhook/subscriber/DuplicateEventsSubscriber.java

src/test/java/org/jenkinsci/plugins/github/admin/GitHubDuplicateEventsMonitorTest.java

…rties Co-authored-by: Jesse Glick <[email protected]>

…teEventsMonitorTest.java Co-authored-by: Jesse Glick <[email protected]>

…teEventsMonitor.java Co-authored-by: Jesse Glick <[email protected]>

…berEvent.java Co-authored-by: Jesse Glick <[email protected]>

… duplicate monitor

gbhat618 · 2025-02-14T07:28:32Z

All updates done after the code reviews for now. I have tested and made sure the same testing notes are still applicable and tested again as well.
I would appreciate any reviews, improvement suggestions. Thank you.

src/main/java/org/jenkinsci/plugins/github/admin/GitHubDuplicateEventsMonitor.java

jglick · 2025-02-20T00:39:13Z

@KostyaSha are you available to review?

…teEventsMonitor.java Co-authored-by: Jesse Glick <[email protected]>

KostyaSha · 2025-02-20T00:42:31Z

@KostyaSha are you available to review?

i trust you, can merge as is.
Also would it be ok if i squash merge it?

gbhat618 · 2025-02-20T01:58:16Z

imo, it should be squash merged

KostyaSha · 2025-02-20T23:50:47Z

[INFO] [ERROR] Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.078 s <<< FAILURE! -- in org.jenkinsci.plugins.github.admin.GitHubDuplicateEventsMonitorUnitTest
[INFO] [ERROR] org.jenkinsci.plugins.github.admin.GitHubDuplicateEventsMonitorUnitTest.checkOldEntriesAreExpiredAfter10Minutes -- Time elapsed: 0.006 s <<< FAILURE!
[INFO] java.lang.AssertionError:
[INFO]
[INFO] Expected: is <[2, 1]>
[INFO]      but: was <[1]>
[INFO] 	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
[INFO] 	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:6)
[INFO] 	at org.jenkinsci.plugins.github.admin.GitHubDuplicateEventsMonitorUnitTest.checkOldEntriesAreExpiredAfter10Minutes(GitHubDuplicateEventsMonitorUnitTest.java:86)
[INFO] 	at java.base/java.lang.reflect.Method.invoke(Method.java:568)

gbhat618 · 2025-02-21T02:36:09Z

The test failure above corresponds to

github-plugin/src/test/java/org/jenkinsci/plugins/github/admin/GitHubDuplicateEventsMonitorUnitTest.java

Lines 83 to 86 in 596fbc3

    
           // at present 
        
           subscriber.onEvent(new GHSubscriberEvent("1", "origin", GHEvent.PUSH, "payload")); 
        
           subscriber.onEvent(new GHSubscriberEvent("2", "origin", GHEvent.PUSH, "payload")); 
        
           assertThat(getEventCountsTracker(), is(Set.of("1", "2")));

This doesn't have any timing logic, so I am getting delayed to figure out what could have caused (even the timing logic shouldn't cause as problem, as a FakeTicker is used to mock time)

I couldn't reproduce in local (terminal (mvn clean test and intellij test runner)

mvn test logs correctly printed the number of tests passed

[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
Running tests for com.coravy.hudson.plugins.github:github:1.41.1-SNAPSHOT
[INFO] Running com.cloudbees.jenkins.GitHubCommitNotifierTest
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.09 s -- in com.cloudbees.jenkins.GitHubCommitNotifierTest
[INFO] Running com.cloudbees.jenkins.GitHubPushTriggerTest
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 15.93 s -- in com.cloudbees.jenkins.GitHubPushTriggerTest
[INFO] Running com.cloudbees.jenkins.GitHubSetCommitStatusBuilderTest
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.396 s -- in com.cloudbees.jenkins.GitHubSetCommitStatusBuilderTest
[INFO] Running com.cloudbees.jenkins.GitHubWebHookCrumbExclusionTest
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.475 s -- in com.cloudbees.jenkins.GitHubWebHookCrumbExclusionTest
[INFO] Running com.cloudbees.jenkins.GitHubWebHookFullTest
[INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.550 s -- in com.cloudbees.jenkins.GitHubWebHookFullTest
[INFO] Running com.cloudbees.jenkins.GitHubWebHookTest
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.217 s -- in com.cloudbees.jenkins.GitHubWebHookTest
[INFO] Running com.cloudbees.jenkins.GlobalConfigSubmitTest
[WARNING] Tests run: 3, Failures: 0, Errors: 0, Skipped: 3, Time elapsed: 0 s -- in com.cloudbees.jenkins.GlobalConfigSubmitTest
[INFO] Running com.coravy.hudson.plugins.github.GitHubRepositoryNameTest
[INFO] Tests run: 56, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.036 s -- in com.coravy.hudson.plugins.github.GitHubRepositoryNameTest
[INFO] Running com.coravy.hudson.plugins.github.GithubLinkActionFactoryTest
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.563 s -- in com.coravy.hudson.plugins.github.GithubLinkActionFactoryTest
[INFO] Running com.coravy.hudson.plugins.github.GithubLinkAnnotatorTest
[INFO] Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.156 s -- in com.coravy.hudson.plugins.github.GithubLinkAnnotatorTest
[INFO] Running com.coravy.hudson.plugins.github.GithubProjectPropertyTest
[WARNING] Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 0 s -- in com.coravy.hudson.plugins.github.GithubProjectPropertyTest
[INFO] Running com.coravy.hudson.plugins.github.GithubUrlTest
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.007 s -- in com.coravy.hudson.plugins.github.GithubUrlTest
[INFO] Running com.coravy.hudson.plugins.github.github.InjectedTest
[INFO] Tests run: 28, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.614 s -- in com.coravy.hudson.plugins.github.github.InjectedTest
[INFO] Running org.jenkinsci.plugins.github.admin.GHRepoNameTest
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.483 s -- in org.jenkinsci.plugins.github.admin.GHRepoNameTest
[INFO] Running org.jenkinsci.plugins.github.admin.GitHubDuplicateEventsMonitorTest
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 14.72 s -- in org.jenkinsci.plugins.github.admin.GitHubDuplicateEventsMonitorTest
[INFO] Running org.jenkinsci.plugins.github.admin.GitHubDuplicateEventsMonitorUnitTest
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.031 s -- in org.jenkinsci.plugins.github.admin.GitHubDuplicateEventsMonitorUnitTest
[INFO] Running org.jenkinsci.plugins.github.admin.GitHubHookRegisterProblemMonitorTest
[INFO] Tests run: 21, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 19.26 s -- in org.jenkinsci.plugins.github.admin.GitHubHookRegisterProblemMonitorTest
[INFO] Running org.jenkinsci.plugins.github.admin.ValidateRepoNameTest
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.520 s -- in org.jenkinsci.plugins.github.admin.ValidateRepoNameTest
[INFO] Running org.jenkinsci.plugins.github.common.CombineErrorHandlerTest
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.710 s -- in org.jenkinsci.plugins.github.common.CombineErrorHandlerTest
[INFO] Running org.jenkinsci.plugins.github.common.ExpandableMessageTest
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.580 s -- in org.jenkinsci.plugins.github.common.ExpandableMessageTest
[INFO] Running org.jenkinsci.plugins.github.config.ConfigAsCodeTest
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.018 s -- in org.jenkinsci.plugins.github.config.ConfigAsCodeTest
[INFO] Running org.jenkinsci.plugins.github.config.GitHubPluginConfigTest
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 17.11 s -- in org.jenkinsci.plugins.github.config.GitHubPluginConfigTest
[INFO] Running org.jenkinsci.plugins.github.config.GitHubServerConfigIntegrationTest
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.367 s -- in org.jenkinsci.plugins.github.config.GitHubServerConfigIntegrationTest
[INFO] Running org.jenkinsci.plugins.github.config.GitHubServerConfigTest
[INFO] Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.053 s -- in org.jenkinsci.plugins.github.config.GitHubServerConfigTest
[INFO] Running org.jenkinsci.plugins.github.config.HookSecretConfigTest
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.370 s -- in org.jenkinsci.plugins.github.config.HookSecretConfigTest
[INFO] Running org.jenkinsci.plugins.github.extension.CryptoUtilTest
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.001 s -- in org.jenkinsci.plugins.github.extension.CryptoUtilTest
[INFO] Running org.jenkinsci.plugins.github.extension.GHEventsSubscriberTest
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.017 s -- in org.jenkinsci.plugins.github.extension.GHEventsSubscriberTest
[INFO] Running org.jenkinsci.plugins.github.internal.GitHubClientCacheCleanupTest
[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.457 s -- in org.jenkinsci.plugins.github.internal.GitHubClientCacheCleanupTest
[INFO] Running org.jenkinsci.plugins.github.internal.GitHubClientCacheOpsTest
[INFO] Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.149 s -- in org.jenkinsci.plugins.github.internal.GitHubClientCacheOpsTest
[INFO] Running org.jenkinsci.plugins.github.migration.MigratorTest
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.893 s -- in org.jenkinsci.plugins.github.migration.MigratorTest
[INFO] Running org.jenkinsci.plugins.github.status.GitHubCommitStatusSetterTest
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.305 s -- in org.jenkinsci.plugins.github.status.GitHubCommitStatusSetterTest
[INFO] Running org.jenkinsci.plugins.github.status.err.ErrorHandlersTest
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.627 s -- in org.jenkinsci.plugins.github.status.err.ErrorHandlersTest
[INFO] Running org.jenkinsci.plugins.github.status.sources.BuildRefBackrefSourceTest
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.430 s -- in org.jenkinsci.plugins.github.status.sources.BuildRefBackrefSourceTest
[INFO] Running org.jenkinsci.plugins.github.status.sources.ConditionalStatusResultSourceTest
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.699 s -- in org.jenkinsci.plugins.github.status.sources.ConditionalStatusResultSourceTest
[INFO] Running org.jenkinsci.plugins.github.status.sources.DefaultStatusResultSourceTest
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.689 s -- in org.jenkinsci.plugins.github.status.sources.DefaultStatusResultSourceTest
[INFO] Running org.jenkinsci.plugins.github.status.sources.ManuallyEnteredRepositorySourceTest
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.663 s -- in org.jenkinsci.plugins.github.status.sources.ManuallyEnteredRepositorySourceTest
[INFO] Running org.jenkinsci.plugins.github.status.sources.ManuallyEnteredSourcesTest
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.680 s -- in org.jenkinsci.plugins.github.status.sources.ManuallyEnteredSourcesTest
[INFO] Running org.jenkinsci.plugins.github.status.sources.misc.AnyBuildResultTest
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.582 s -- in org.jenkinsci.plugins.github.status.sources.misc.AnyBuildResultTest
[INFO] Running org.jenkinsci.plugins.github.status.sources.misc.BetterThanOrEqualBuildResultTest
[INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.601 s -- in org.jenkinsci.plugins.github.status.sources.misc.BetterThanOrEqualBuildResultTest
[INFO] Running org.jenkinsci.plugins.github.util.BuildDataHelperTest
[INFO] Running org.jenkinsci.plugins.github.util.BuildDataHelperTest$WhenBuildingOrganizationJobs
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.036 s -- in org.jenkinsci.plugins.github.util.BuildDataHelperTest$WhenBuildingOrganizationJobs
[INFO] Running org.jenkinsci.plugins.github.util.BuildDataHelperTest$WhenBuildingRegularJobs
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.002 s -- in org.jenkinsci.plugins.github.util.BuildDataHelperTest$WhenBuildingRegularJobs
[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.045 s -- in org.jenkinsci.plugins.github.util.BuildDataHelperTest
[INFO] Running org.jenkinsci.plugins.github.util.JobInfoHelpersTest
[INFO] Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.057 s -- in org.jenkinsci.plugins.github.util.JobInfoHelpersTest
[INFO] Running org.jenkinsci.plugins.github.util.XSSApiTest
[INFO] Tests run: 15, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.015 s -- in org.jenkinsci.plugins.github.util.XSSApiTest
[INFO] Running org.jenkinsci.plugins.github.webhook.GHEventHeaderTest
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.486 s -- in org.jenkinsci.plugins.github.webhook.GHEventHeaderTest
[INFO] Running org.jenkinsci.plugins.github.webhook.GHEventPayloadTest
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.496 s -- in org.jenkinsci.plugins.github.webhook.GHEventPayloadTest
[INFO] Running org.jenkinsci.plugins.github.webhook.RequirePostWithGHHookPayloadTest
[INFO] Tests run: 16, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 14.52 s -- in org.jenkinsci.plugins.github.webhook.RequirePostWithGHHookPayloadTest
[INFO] Running org.jenkinsci.plugins.github.webhook.WebhookManagerTest
[INFO] Tests run: 16, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.37 s -- in org.jenkinsci.plugins.github.webhook.WebhookManagerTest
[INFO] Running org.jenkinsci.plugins.github.webhook.subscriber.DefaultPushGHEventListenerTest
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.270 s -- in org.jenkinsci.plugins.github.webhook.subscriber.DefaultPushGHEventListenerTest
[INFO] Running org.jenkinsci.plugins.github.webhook.subscriber.PingGHEventSubscriberTest
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.031 s -- in org.jenkinsci.plugins.github.webhook.subscriber.PingGHEventSubscriberTest
[INFO]
[INFO] Results:
[INFO]
[WARNING] Tests run: 315, Failures: 0, Errors: 0, Skipped: 4
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------

gbhat618 · 2025-02-21T02:36:57Z

did it happen all the time or was it flaky ?

(mean time I am trying to think more and see if I can reproduce)

KostyaSha · 2025-02-21T17:31:27Z

I run twice during release process and it failed both times...

jglick · 2025-02-21T20:11:41Z

Passing for me on Ubuntu with Java 21.

KostyaSha · 2025-02-21T22:15:54Z

released with 21, 17 failed before

gbhat618 · 2025-02-23T08:20:10Z

I tested with Java 17 but couldn't reproduce the issue, so it's likely not related to the Java version.

Instead, it might be due to the static nature of the cache:

github-plugin/src/main/java/org/jenkinsci/plugins/github/admin/GitHubDuplicateEventsMonitor.java

Lines 117 to 121 in 54fb24f

    
           private static final Cache<String, Object> EVENT_TRACKER = Caffeine.newBuilder() 
        
                                                                              .maximumSize(24_000L) 
        
                                                                              .expireAfterWrite(Duration.ofMinutes(10)) 
        
                                                                              .ticker(() -> ticker.read()) 
        
                                                                              .build();

If the two tests in GitHubDuplicateEventsMonitorUnitTest run in parallel, they may interfere with cache expiration by advancing the fakeTicker. However, still not sure why 2 would be missing—if it were 1, that would be more explainable.

[INFO] Expected: is <[2, 1]>
[INFO]      but: was <[1]>

Caffeine is already thread-safe, so it’s unlikely to be causing racing condition in itself during parallel executions.

gbhat618 · 2025-02-23T08:28:30Z

To prevent cache residues, I can add an @Before method to clear the cache. However, the tests should still run sequentially since the cache is a static variable.

Are the tests already running sequentially (in local they seem to run sequential), or are they parallelized?
I didn't find in plugin pom and parent pom - but I am not sure if it is configured in other ways or effectively inherited in some way

jglick · 2025-02-24T17:13:32Z

Ah. @gbhat618

github-plugin/src/main/java/org/jenkinsci/plugins/github/admin/GitHubDuplicateEventsMonitor.java

Line 117 in 54fb24f

    
           private static final Cache<String, Object> EVENT_TRACKER = Caffeine.newBuilder()

is wrong; remove static. (It should be an instance field of the @Extension singleton.) Usages outside instance methods of the class should therefore use ExtensionList.lookupSingleton.

In general, this mistake can cause hard-to-track test failures when using JenkinsRule (or JenkinsSessionRule), though not RealJenkinsRule, since different test cases (methods) and even suites (classes) could be using the same java.lang.Class in different Jenkins sessions simultaneously.

gbhat618 · 2025-02-25T02:12:50Z

noted about it. thank you.

sorry for the mistake on this,
here created the fix PR #389

gbhat618 added 2 commits January 31, 2025 21:04

create an admin monitor for warning about duplicate events from github

fc6e29c

update the list of interested events and minor changes in cleanup

a8bcb47

gbhat618 mentioned this pull request Feb 3, 2025

Create an admin monitor for warning about duplicate events from GitHub gbhat618/github-plugin#1

Closed

8 tasks

gbhat618 added 3 commits February 4, 2025 10:08

consider tests sending stapler currentRequest null

3d9c96b

use mocks instead of putting null check in execution code

0a3820c

remove the fixed comment

c8f5a01

Vlatombe approved these changes Feb 4, 2025

View reviewed changes

gbhat618 and others added 3 commits February 5, 2025 23:33

track the last seen duplicate for 24 hours

978b8b6

Update src/main/java/org/jenkinsci/plugins/github/webhook/subscriber/…

bfd9df8

…DuplicateEventsSubscriber.java Co-authored-by: Vincent Latombe <[email protected]>

remove few bits to further simplify

e77068d

gbhat618 marked this pull request as ready for review February 6, 2025 04:58

gbhat618 requested a review from a team as a code owner February 6, 2025 04:58

gbhat618 added 4 commits February 6, 2025 15:18

provide a way to extract last logged event via logging in duplicates …

188be21

…admin monitor (checking if this is the best, if yes then will write a test for it)

add a comment for why tracking the prev event inside the logger eval

89ef2bd

log message should be clear to the user

e3d9f80

update the admin monitor test to include duplicate event logging logic

a5dd3bf

gbhat618 marked this pull request as draft February 7, 2025 04:13

show the payload via hyperlink not logging

b5ab212

github-advanced-security bot found potential problems Feb 7, 2025

View reviewed changes

src/main/java/org/jenkinsci/plugins/github/admin/GitHubDuplicateEventsMonitor.java Fixed Show resolved Hide resolved

mark the method with GET

57f050f

gbhat618 marked this pull request as ready for review February 7, 2025 16:38

use variable for shorter line

c0fe850

jglick reviewed Feb 10, 2025

View reviewed changes

src/main/java/org/jenkinsci/plugins/github/extension/GHSubscriberEvent.java Outdated Show resolved Hide resolved

jglick reviewed Feb 10, 2025

View reviewed changes

src/main/java/org/jenkinsci/plugins/github/webhook/subscriber/DuplicateEventsSubscriber.java Outdated Show resolved Hide resolved

jglick reviewed Feb 10, 2025

View reviewed changes

src/main/java/org/jenkinsci/plugins/github/webhook/subscriber/DuplicateEventsSubscriber.java Outdated Show resolved Hide resolved

jglick reviewed Feb 10, 2025

View reviewed changes

src/test/java/org/jenkinsci/plugins/github/admin/GitHubDuplicateEventsMonitorTest.java Outdated Show resolved Hide resolved

gbhat618 and others added 10 commits February 11, 2025 06:01

Update src/main/resources/org/jenkinsci/plugins/github/Messages.prope…

3e75413

…rties Co-authored-by: Jesse Glick <[email protected]>

Update src/test/java/org/jenkinsci/plugins/github/admin/GitHubDuplica…

e8fbb20

…teEventsMonitorTest.java Co-authored-by: Jesse Glick <[email protected]>

Update src/main/java/org/jenkinsci/plugins/github/admin/GitHubDuplica…

39350c7

…teEventsMonitor.java Co-authored-by: Jesse Glick <[email protected]>

Update src/main/java/org/jenkinsci/plugins/github/admin/GitHubDuplica…

5577148

…teEventsMonitor.java Co-authored-by: Jesse Glick <[email protected]>

update for comments

8473753

add a comment and fix the javadoc

e46168d

Update src/main/java/org/jenkinsci/plugins/github/extension/GHSubscri…

626e368

…berEvent.java Co-authored-by: Jesse Glick <[email protected]>

use caffeine for cache instead of ConcurrentHashMap

c65fc31

add a note about cache size

8334bda

move the duplicate event subscriber as an inner class into the github…

82c5ab6

… duplicate monitor

jglick reviewed Feb 20, 2025

View reviewed changes

src/main/java/org/jenkinsci/plugins/github/admin/GitHubDuplicateEventsMonitor.java Outdated Show resolved Hide resolved

jglick approved these changes Feb 20, 2025

View reviewed changes

Update src/main/java/org/jenkinsci/plugins/github/admin/GitHubDuplica…

50108ef

…teEventsMonitor.java Co-authored-by: Jesse Glick <[email protected]>

KostyaSha merged commit 596fbc3 into jenkinsci:master Feb 20, 2025
17 checks passed

gbhat618 deleted the admin-monitor-for-duplicate-events branch February 20, 2025 02:53

gbhat618 mentioned this pull request Feb 25, 2025

Do not use static state variables in Extension class #389

Open

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Warn about duplicated events received from GitHub via Admin Monitor #388

Warn about duplicated events received from GitHub via Admin Monitor #388

gbhat618 commented Feb 3, 2025 •

edited

Loading

gbhat618 commented Feb 3, 2025

Vlatombe left a comment

gbhat618 commented Feb 7, 2025 •

edited

Loading

gbhat618 commented Feb 7, 2025

gbhat618 commented Feb 10, 2025

gbhat618 commented Feb 14, 2025 •

edited

Loading

jglick commented Feb 20, 2025

KostyaSha commented Feb 20, 2025 •

edited

Loading

gbhat618 commented Feb 20, 2025

KostyaSha commented Feb 20, 2025

gbhat618 commented Feb 21, 2025

gbhat618 commented Feb 21, 2025

KostyaSha commented Feb 21, 2025

jglick commented Feb 21, 2025

KostyaSha commented Feb 21, 2025

gbhat618 commented Feb 23, 2025

gbhat618 commented Feb 23, 2025 •

edited

Loading

jglick commented Feb 24, 2025 •

edited

Loading

gbhat618 commented Feb 25, 2025

Warn about duplicated events received from GitHub via Admin Monitor #388

Warn about duplicated events received from GitHub via Admin Monitor #388

Conversation

gbhat618 commented Feb 3, 2025 • edited Loading

Testing done

Submitter checklist

gbhat618 commented Feb 3, 2025

Vlatombe left a comment

Choose a reason for hiding this comment

gbhat618 commented Feb 7, 2025 • edited Loading

gbhat618 commented Feb 7, 2025

gbhat618 commented Feb 10, 2025

gbhat618 commented Feb 14, 2025 • edited Loading

jglick commented Feb 20, 2025

KostyaSha commented Feb 20, 2025 • edited Loading

gbhat618 commented Feb 20, 2025

KostyaSha commented Feb 20, 2025

gbhat618 commented Feb 21, 2025

gbhat618 commented Feb 21, 2025

KostyaSha commented Feb 21, 2025

jglick commented Feb 21, 2025

KostyaSha commented Feb 21, 2025

gbhat618 commented Feb 23, 2025

gbhat618 commented Feb 23, 2025 • edited Loading

jglick commented Feb 24, 2025 • edited Loading

gbhat618 commented Feb 25, 2025

gbhat618 commented Feb 3, 2025 •

edited

Loading

gbhat618 commented Feb 7, 2025 •

edited

Loading

gbhat618 commented Feb 14, 2025 •

edited

Loading

KostyaSha commented Feb 20, 2025 •

edited

Loading

gbhat618 commented Feb 23, 2025 •

edited

Loading

jglick commented Feb 24, 2025 •

edited

Loading