Panic: send on closed channel when closing database

[sethrosetter]

I saw this test functional_tests.test_db_online_offline_resync.test_bucket_online_offline_resync_sanity[10-100-5] fail during this run. http://qa.sc.couchbase.com/view/mobile/job/centos-syncgateway-functional-tests/112/
Upon inspecting the logs, I noticed the below panic.

Using 1.2.0-77 with CBS 4.1.0. It looks go-couchbase related.

2016-02-01T18:07:22.711-08:00 HTTP+: #001:     --> 200   (1514.7 ms)
2016-02-01T18:07:22.717-08:00 HTTP:  #002: POST /db/_online  (ADMIN)
2016-02-01T18:07:22.717-08:00 CRUD: Taking Database : db, online in 0 seconds
2016-02-01T18:07:22.717-08:00 HTTP+: #002:     --> 200   (0.1 ms)
2016-02-01T18:07:22.717-08:00 Closing db /db (bucket "data-bucket")
2016-02-01T18:07:22.717-08:00 Opening db /db as bucket "data-bucket", pool "default", server <http://172.23.106.39:8091>
panic: send on closed channel

sync_gateway logs - https://gist.github.com/sethrosetter/bc3a171a3583e5f47595#file-sync_gateway_error-log-L2552-L2561

functional test output - https://gist.github.com/sethrosetter/c7dae515492169067488

Possibly related to or a duplicate of this - couchbase/go-couchbase#72

[sethrosetter]

Reran the test and it passed - http://qa.sc.couchbase.com/view/mobile/job/centos-syncgateway-functional-tests/113/

[sethrosetter]

Going to rerun in a loop of 50 and see if it fails - http://qa.sc.couchbase.com/view/mobile/job/centos-syncgateway-functional-tests/114/

[zgramana]

Given that it's intermittent, and a possible dup, I'm tagging as 1.3 + hotfix. Expect to close once we confirm that it's definitely a duplicate.

[sethrosetter]

@adamcfraser per the run of 50 above, it ran successfully 6 times and failed on the 7th - http://qa.sc.couchbase.com/view/mobile/job/centos-syncgateway-functional-tests/114/

[ajres]

I think the underlying issue here is that the TapFeed.output channel is being closed by the reader goroutine not the writer goroutine. This means the writer has no knowledge about the output channel state, so if it is processing an event when the output channel is closed by the reader it will try to write to that channel and panic.

Instead of calling close(feed.output) in the TapFeed.Close() method (called by reader), we could call close(feed.output) when the feed.quit channel is closed and returns in the main select loop:

for { select { case event, ok := <-singleFeed.C: if !ok { if singleFeed.Error != nil { log.Printf("go-couchbase: Tap feed from %s failed: %v", host, singleFeed.Error) } killSwitch <- true return } feed.output <- event case <-feed.quit:close(feed.output)return } }

[zgramana]

@ajres will review the Gerrit ticket this week.

[ajres]

gerrit ticket has been opened

NOTE: This ticket was abandoned after local repo was lost and could not retrieve code from gerrit.

[ajres]

Add SG unit and functional test

[ajres]

new gerrit ticket has been opened with original review feedback

[adamcfraser]

Merged to go-couchbase, just need to update SG manifest to pick it up.

[adamcfraser]

Picked up with #2041