runtime: concurrent GC causes SIGBUS, SIGSEGV

[robx]

We experimented with current master, and are experiencing regular crashes under reasonably high load. I haven't been able to reduce this yet, but maybe it's helpful as is? Let me know if there's something else I can provide.

$ go version
go version devel +8b186df Thu May 28 02:30:26 2015 +0000 darwin/amd64

Cross-compiled to linux/amd64.

unexpected fault address 0xc2126257a0
fatal error: fault
[signal 0xb code=0x2 addr=0xc2126257a0 pc=0xc2126257a0]

goroutine 31 [running]:
runtime.throw(0xad7840, 0x5)
        /s/go/src/runtime/panic.go:527 +0x96 fp=0xc20f3eb8c0 sp=0xc20f3eb8a8
runtime.sigpanic()
        /s/go/src/runtime/sigpanic_unix.go:27 +0x2ae fp=0xc20f3eb910 sp=0xc20f3eb8c0
created by realzeit.io/rtb/decision-engine/server.NewCampaign
        /go/src/realzeit.io/rtb/decision-engine/server/campaign.go:40 +0x15e

goroutine 1 [select, 1 minutes]:
main.main()
        /go/src/realzeit.io/rtb/one/main.go:117 +0x10e0

...

The relevant code:

    28  func NewCampaign(de *bidding.DecisionEngine) *campaign {
    29      c := campaign{
    30          de:         de,
    31          requests:   make(chan crequest),
    32          updCamp:    make(chan rzcampaign.Config),
    33          updBrain:   make(chan rzcampaign.BrainConfig),
    34          debugStats: make(chan chan DebugStats),
    35      }
    36      c.wg.Add(1)
    37      go func() {
    38          c.loop()
    39          c.wg.Done()
    40      }()
    41      return &c
    42  }

[davecheney]

Have you run your application under the replace detector?

[minux]

Does the program use cgo? unsafe? Custom assembly functions? Have you tried running it with the race detector? What's the full stack trace?

[robx]

We haven't run it with -race under proper load, will do that once I get it compiled (currently stuck on #9918). No complaints in tests. (I assume "replace" above means "race".)

It does not use cgo. The only use of unsafe is via github.com/oschwald/maxminddb-golang (https://github.com/oschwald/maxminddb-golang/blob/master/key_other.go). There's no assembler in there.

The full stack trace is at https://gist.github.com/robx/ee71a5868ff136f04aab

[robx]

-race doesn't complain, here's three more stack traces, thanks @alicebob.

go version devel +80864cf Fri May 29 04:33:15 2015 +0000 linux/amd64

unexpected fault address 0x0
fatal error: fault
[signal 0xb code=0x80 addr=0x0 pc=0x589e61]

goroutine 132 [running]:
runtime.throw(0xcb4a60, 0x5)
        /home/harmen/src/go/src/runtime/panic.go:527 +0x96 fp=0xc20f4d58c8 sp=0xc20f4d58b0
runtime.sigpanic()
        /home/harmen/src/go/src/runtime/sigpanic_unix.go:27 +0x2ae fp=0xc20f4d5918 sp=0xc20f4d58c8
realzeit.io/rtb/decision-engine/server.(*campaign).loop(0xc2082caa80)
        /home/harmen/play/src/realzeit.io/rtb/decision-engine/server/campaign.go:51 +0x231 fp=0xc20f4d5fc8 sp=0
xc20f4d5918
realzeit.io/rtb/decision-engine/server.NewCampaign.func1(0xc2082caa80)
        /home/harmen/play/src/realzeit.io/rtb/decision-engine/server/campaign.go:38 +0x36 fp=0xc20f4d5fd8 sp=0x
c20f4d5fc8
runtime.goexit()
        /home/harmen/src/go/src/runtime/asm_amd64.s:1670 +0x1 fp=0xc20f4d5fe0 sp=0xc20f4d5fd8
created by realzeit.io/rtb/decision-engine/server.NewCampaign
        /home/harmen/play/src/realzeit.io/rtb/decision-engine/server/campaign.go:40 +0x183

goroutine 1 [select]:
main.main()
        /home/harmen/play/src/realzeit.io/rtb/one/main.go:117 +0x1404

unexpected fault address 0xc2132024e0
fatal error: fault
[signal 0xb code=0x2 addr=0xc2132024e0 pc=0xc2132024e0]

goroutine 57 [running]:
runtime.throw(0xcb4a60, 0x5)
        /home/harmen/src/go/src/runtime/panic.go:527 +0x96 fp=0xc20e5478c0 sp=0xc20e5478a8
runtime.sigpanic()
        /home/harmen/src/go/src/runtime/sigpanic_unix.go:27 +0x2ae fp=0xc20e547910 sp=0xc20e5478c0
created by realzeit.io/rtb/decision-engine/server.NewCampaign
        /home/harmen/play/src/realzeit.io/rtb/decision-engine/server/campaign.go:40 +0x183

goroutine 1 [select]:
main.main()
        /home/harmen/play/src/realzeit.io/rtb/one/main.go:117 +0x1404

goroutine 17 [syscall, locked to thread]:
runtime.goexit()
        /home/harmen/src/go/src/runtime/asm_amd64.s:1670 +0x1

unexpected fault address 0xc20edef460
fatal error: fault
[signal 0xb code=0x2 addr=0xc20edef460 pc=0xc20edef460]

goroutine 197 [running]:
runtime.throw(0xcb4a60, 0x5)
        /home/harmen/src/go/src/runtime/panic.go:527 +0x96 fp=0xc2119518c0 sp=0xc2119518a8
runtime.sigpanic()
        /home/harmen/src/go/src/runtime/sigpanic_unix.go:27 +0x2ae fp=0xc211951910 sp=0xc2119518c0
created by realzeit.io/rtb/decision-engine/server.NewCampaign
        /home/harmen/play/src/realzeit.io/rtb/decision-engine/server/campaign.go:40 +0x183

goroutine 1 [select, 1 minutes]:
main.main()
        /home/harmen/play/src/realzeit.io/rtb/one/main.go:117 +0x1404

[mikioh]

Can you provide a minimal reproducible snippet for us? The dying messages clearly said that it went wrong in (*campaign).loop(); your code, runtime or whatever touched non-permitted/unmapped stuff on addressing space.

[minux]

On Fri, May 29, 2015 at 5:53 AM, Robert notifications@github.com wrote:

go version devel +80864cf Fri May 29 04:33:15 2015 +0000 linux/amd64

unexpected fault address 0x0
fatal error: fault
[signal 0xb code=0x80 addr=0x0 pc=0x589e61]

This fault is interesting. Note the code is 0x80, normally, for SIGSEGV,
the code
should be 0/1/2 for user triggered fault. 0x80 means it's sent from kernel.

I don't know why would kernel send a synchronous signal like SIGSEGV.

[mikioh]

Perhaps 0x80 comes from:

/usr/include/sys/wait.h:
#define WCOREFLAG   0200

[robx]

I'll try boiling it down.

Note that the crash is on linux/amd64, it's just that the first crashing version was cross-compiled from darwin/amd64. The second one (with -race) was built on linux/amd64 directly.

[robx]

There's rather reduced test case in https://github.com/robx/gocrash. It reliably crashes for me when running

go test -bench=. -cpu=2

on OS X, go version devel +8b186df Thu May 28 02:30:26 2015 +0000 darwin/amd64.

It's likely that there's more that can be removed, but I've tried a couple of things that I tried towards the end seemed to be necessary to cause the problems.

[davecheney]

This part of the test makes no sense reps := make(chan Response, 1) r.rep <- reps this will always block the goroutine with no possibility of waking it up.

[mikioh]

Thanks for the gocrash. I just saw the crash w/ tip on freebsd/amd64.

fatal error: fault
[signal 0xa code=0xc addr=0xc20a610a40 pc=0xc20a610a40]

goroutine 16371 [running]:
runtime.throw(0x583510, 0x5)
    /go/src/runtime/panic.go:527 +0x96 fp=0xc2083d9e28 sp=0xc2083d9e10
runtime.sigpanic()
    /go/src/runtime/sigpanic_unix.go:21 +0x1e7 fp=0xc2083d9e78 sp=0xc2083d9e28
created by _/home/mikioh/gocrash.NewHandler
    /gocrash/handler.go:25 +0xd5

and

TEXT _/gocrash.NewHandler(SB) /gocrash/handler.go
        handler.go:17   0x476d80        64488b0c25f8ffffff      FS MOVQ FS:0xfffffff8, CX
        handler.go:17   0x476d89        483b6110                CMPQ 0x10(CX), SP
(snip)
        handler.go:25   0x476e4b        4889442408              MOVQ AX, 0x8(SP)
        handler.go:25   0x476e50        e8bb82fbff              CALL runtime.newproc(SB)
        handler.go:26   0x476e55        488b5c2418              MOVQ 0x18(SP), BX
        handler.go:26   0x476e5a        48895c2448              MOVQ BX, 0x48(SP)

I guess this is a memory corruption issue by new garbage collector. If you never see this sort of crash when you run your software with GOGC=off, it's likely.

[robx]

@davecheney I don't think so? manager.Handle receives from r.rep as far as I can tell. But probably I misunderstand.

I agree that the code is a bit weird right now, it used to do more (particularly collect responses from multiple handlers).