runtime: GOEXPERIMENT=precisestack GOGC=0 ./all.bash fails

[randall77]

We're getting occasional failures in the garbage collector when it tries to scan an
interface and gets a bad interface table pointer.

sync to tip
set GOEXPERIMENT=precisestack
set GOGC=0
run all.bash

With GOGC=0 it crashes reliably.

SIGSEGV: segmentation violation
PC=0x46daa4

scaninterfacedata(0x2, 0x7f7c76899290, 0x1, 0x7f7c769b5dc0)
    /usr/local/google/home/khr/sandbox/go-ro2/src/pkg/runtime/mgc0.c:1428 +0x84
scanbitvector(0x7f7c76899290, 0x732f0c, 0x4c4501, 0x7f7c769b5dc0)
    /usr/local/google/home/khr/sandbox/go-ro2/src/pkg/runtime/mgc0.c:1461 +0x11e
scanframe(0x7f7c769b5c50, 0x7f7c769b5dc0)
    /usr/local/google/home/khr/sandbox/go-ro2/src/pkg/runtime/mgc0.c:1525 +0x16e
runtime.gentraceback(0x46eed2, 0x7f7c76899108, 0x0, 0xc20812eb40, 0x0, ...)
    /usr/local/google/home/khr/sandbox/go-ro2/src/pkg/runtime/traceback_x86.c:155 +0x3ff
addstackroots(0xc20812eb40, 0x7f7c769b5dc0)
    /usr/local/google/home/khr/sandbox/go-ro2/src/pkg/runtime/mgc0.c:1581 +0x168
markroot(0xc20802e000, 0x9)
    /usr/local/google/home/khr/sandbox/go-ro2/src/pkg/runtime/mgc0.c:1278 +0xbe
runtime.parfordo(0xc20802e000)
    /usr/local/google/home/khr/sandbox/go-ro2/src/pkg/runtime/parfor.c:105 +0xa3
gc(0x7f7c76899130)
    /usr/local/google/home/khr/sandbox/go-ro2/src/pkg/runtime/mgc0.c:2170 +0x259
mgc(0xc20812eb40)
    /usr/local/google/home/khr/sandbox/go-ro2/src/pkg/runtime/mgc0.c:2111 +0x2e
runtime.mcall(0x47e1f7)
    /usr/local/google/home/khr/sandbox/go-ro2/src/pkg/runtime/asm_amd64.s:181 +0x4b

goroutine 21 [garbage collection]:
runtime.gc(0xc200000000)
    /usr/local/google/home/khr/sandbox/go-ro2/src/pkg/runtime/mgc0.c:2082 +0x1b2 fp=0x7f7c76899140
runtime.mallocgc(0x8, 0x66df00, 0x0)
    /usr/local/google/home/khr/sandbox/go-ro2/src/pkg/runtime/malloc.goc:218 +0x1b8 fp=0x7f7c768991b8
runtime.new(0x66df00, 0x47f6c8)
    /usr/local/google/home/khr/sandbox/go-ro2/src/pkg/runtime/malloc.goc:815 +0x32 fp=0x7f7c768991d8
os/exec.(*Cmd).stdin(0xc208165040, 0xc2080561c8, 0xc208087600, 0x3d)
    /usr/local/google/home/khr/sandbox/go-ro2/src/pkg/os/exec/exec.go:148 +0x40 fp=0x7f7c76899280
...

It is dying here:

            tab = *(Itab**)scanp;
            if(tab->type->size <= sizeof(void*) && (tab->type->kind & KindNoPointers))
                return;

  46da9d:   48 8b 0a                mov    (%rdx),%rcx     ; tab = *(Itab**)scanp;
  46daa0:   48 8b 41 08             mov    0x8(%rcx),%rax  ; tab->type
  46daa4:   48 8b 00                mov    (%rax),%rax     ; tab->type->size --- faults here
  46daa7:   48 83 f8 08             cmp    $0x8,%rax
  46daab:   77 a2                   ja     46da4f <scaninterfacedata+0x2f>

rax is junk (0x672f6c61636f6c2f) at the faulting instruction.  rcx is 0xc208087600,
which is apparently not an *Itab like it should be.

Possibly related to issue #7205.

[randall77]

Comment 2:

This seems to be fixed with the 7229 fix.

Status changed to Duplicate.

Merged into issue #7229.

[randall77]

Comment 3:

Whoops, spoke too soon (GOGC=0 is slow!).  Still happening...

Status changed to New.

[rsc]

Comment 4:

I suspect this is the same as 7205.

[rsc]

Comment 5:

I have squashed a plethora of bugs in the liveness code. It is still not perfect - there
are some errors with the race detector enabled - but I think it should be significantly
better. I am running 
GOEXPERIMENT=precisestack GOGC=0 ./all.bash
overnight and we will see how it does.

Status changed to Accepted.

[rsc]

Comment 6:

The crash in scaninterfacedata appears to be gone. However, it seems to have been
replaced by some dangling pointers. (There are also a handful of timeouts. As you say,
GOGC=0 is slow!) Still, progress I think.
--- FAIL: TestReader (0.09 seconds)
    reader_test.go:225: test 5, entry 0: Incorrect header:
        have {Name:small.txt Mode:420 Uid:1000 Gid:10 Size:5 ModTime:2013-12-03 05:16:10 -0500 EST Typeflag:48 Linkname: Uname:alex Gname:wheel Devmajor:0 Devminor:0 AccessTime:0001-01-01 00:00:00 +0000 UTC ChangeTime:0001-01-01 00:00:00 +0000 UTC Xattrs:map[user.key2:value2 security.selinux:unconfined_u:object_r:default_t:s0]}
        want {Name:small.txt Mode:420 Uid:1000 Gid:10 Size:5 ModTime:2013-12-03 05:16:10.44825232 -0500 EST Typeflag:48 Linkname: Uname:alex Gname:wheel Devmajor:0 Devminor:0 AccessTime:2014-01-15 05:49:51.41987522 -0500 EST ChangeTime:2014-01-15 05:49:16.794414986 -0500 EST Xattrs:map[user.key2:value2 security.selinux:unconfined_u:object_r:default_t:s0 user.key:value]}
    reader_test.go:225: test 5, entry 1: Incorrect header:
        have {Name:small2.txt Mode:420 Uid:1000 Gid:10 Size:11 ModTime:2013-12-03 05:16:10.449252304 -0500 EST Typeflag:48 Linkname: Uname:alex Gname:wheel Devmajor:0 Devminor:0 AccessTime:0001-01-01 00:00:00 +0000 UTC ChangeTime:2013-12-03 05:16:10.449252304 -0500 EST Xattrs:map[security.selinux:unconfined_u:object_r:default_t:s0]}
        want {Name:small2.txt Mode:420 Uid:1000 Gid:10 Size:11 ModTime:2013-12-03 05:16:10.449252304 -0500 EST Typeflag:48 Linkname: Uname:alex Gname:wheel Devmajor:0 Devminor:0 AccessTime:2014-01-15 05:49:51.41987522 -0500 EST ChangeTime:2013-12-03 05:16:10.449252304 -0500 EST Xattrs:map[security.selinux:unconfined_u:object_r:default_t:s0]}
--- FAIL: TestParsePAXTime (0.00 seconds)
    reader_test.go:397: strconv.ParseInt: parsing "0\x00\x00\x00\x00\x00\x00\x000": invalid syntax
FAIL
FAIL    archive/tar 1.050s
--- FAIL: TestTypedefsELF (0.41 seconds)
    type_test.go:108: t_my_struct:
            have �@��ޭޭޭ� {vi volatile int@0; x char@4 : 1@7; y int@4 : 4@27; z [0]int@8; array [40]long long int@8; zz [0]int@328}
            want struct my_struct {vi volatile int@0; x char@4 : 1@7; y int@4 : 4@27; z [0]int@8; array [40]long long int@8; zz [0]int@328}
    type_test.go:108: t_my_struct1:
            have ��C��ޭޭޭ�1 {zz [1]int@0}
            want struct my_struct1 {zz [1]int@0}
    type_test.go:108: t_my_union:
            have `cA��ޭޭޭ�vi volatile int@0; x char@0 : 1@7; y int@0 : 4@28; array [40]long long int@0}
            want union my_union {vi volatile int@0; x char@0 : 1@7; y int@0 : 4@28; array [40]long long int@0}
    type_test.go:108: t_my_enum:
            have @�>��ޭޭޭ�=1; e2=2; e3=-5; e4=1000000000000000}
            want enum my_enum {e1=1; e2=2; e3=-5; e4=1000000000000000}
    type_test.go:108: t_my_list:
            have ��>��ޭޭޭ� short int@0; next *t_my_list@8}
            want struct list {val short int@0; next *t_my_list@8}
    type_test.go:108: t_my_tree:
            have rA��ޭޭޭ�t *struct tree@0; right *struct tree@8; val long long unsigned int@16}
            want struct tree {left *struct tree@0; right *struct tree@8; val long long unsigned int@16}
--- FAIL: TestTypedefsMachO (0.36 seconds)
    type_test.go:108: t_my_struct:
            have ��@��ޭޭޭ� {vi volatile int@0; x char@4 : 1@7; y int@4 : 4@27; z [0]int@8; array [40]long long int@8; zz [0]int@328}
            want struct my_struct {vi volatile int@0; x char@4 : 1@7; y int@4 : 4@27; z [0]int@8; array [40]long long int@8; zz [0]int@328}
    type_test.go:108: t_my_struct1:
            have  �C��ޭޭޭ�1 {zz [1]int@0}
            want struct my_struct1 {zz [1]int@0}
    type_test.go:108: t_my_union:
            have @bA��ޭޭޭ�vi volatile int@0; x char@0 : 1@7; y int@0 : 4@28; array [40]long long int@0}
            want union my_union {vi volatile int@0; x char@0 : 1@7; y int@0 : 4@28; array [40]long long int@0}
    type_test.go:108: t_my_enum:
            have ��>��ޭޭޭ�=1; e2=2; e3=-5; e4=-1530494976}
            want enum my_enum {e1=1; e2=2; e3=-5; e4=1000000000000000}
    type_test.go:108: t_my_list:
            have ��>��ޭޭޭ� short int@0; next *t_my_list@8}
            want struct list {val short int@0; next *t_my_list@8}
    type_test.go:108: t_my_tree:
            have @tA��ޭޭޭ�t *struct tree@0; right *struct tree@8; val long long unsigned int@16}
            want struct tree {left *struct tree@0; right *struct tree@8; val long long unsigned int@16}
--- FAIL: TestTypedefsELFDwarf4 (0.96 seconds)
    type_test.go:108: t_my_struct:
            have ��@��ޭޭޭ� {vi volatile int@0; x char@4 : 1@7; y int@4 : 4@27; z [0]int@8; array [40]long long int@8; zz [0]int@328}
            want struct my_struct {vi volatile int@0; x char@4 : 1@7; y int@4 : 4@27; z [0]int@8; array [40]long long int@8; zz [0]int@328}
    type_test.go:108: t_my_struct1:
            have ��C��ޭޭޭ�1 {zz [1]int@0}
            want struct my_struct1 {zz [1]int@0}
    type_test.go:108: t_my_union:
            have �cA��ޭޭޭ�vi volatile int@0; x char@0 : 1@7; y int@0 : 4@28; array [40]long long int@0}
            want union my_union {vi volatile int@0; x char@0 : 1@7; y int@0 : 4@28; array [40]long long int@0}
    type_test.go:108: t_my_enum:
            have p�>��ޭޭޭ�=1; e2=2; e3=-5; e4=1000000000000000}
            want enum my_enum {e1=1; e2=2; e3=-5; e4=1000000000000000}
    type_test.go:108: t_my_list:
            have �>��ޭޭޭ� short int@0; next *t_my_list@8}
            want struct list {val short int@0; next *t_my_list@8}
    type_test.go:108: t_my_tree:
            have `cA��ޭޭޭ�t *struct tree@0; right *struct tree@8; val long long unsigned int@16}
            want struct tree {left *struct tree@0; right *struct tree@8; val long long unsigned int@16}
FAIL
FAIL    debug/dwarf 2.131s
--- FAIL: TestNames (0.01 seconds)
    elf_test.go:46: #7: want "+0x50\x00\x00\x00\x00+0x50" have "PF_W+PF_R+0x50"
FAIL
FAIL    debug/elf   1.494s
--- FAIL: TestObjectIdentifier (0.01 seconds)
    asn1_test.go:237: bad ObjectIdentifier.String(). Got 4, want 1.2.3.4
FAIL
FAIL    encoding/asn1   0.822s
--- FAIL: TestStructType (0.04 seconds)
    type_test.go:152: struct printed as "00\x03\b\xc2\x00\x00\x00\xadޭޭޭ\xdeb!\f\x00\x00\x00\x00\x00\xe0\x9f%\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00D bytes; E float; F float; G \x90$ \x00\x00\x00\x00\x00\x06\x00\x00\x00\x00\x00\x00X string; }; H Bar; I Foo; }"; expected "Foo = struct { A int; B int; C string; D bytes; E float; F float; G Bar = struct { X string; }; H Bar; I Foo; }"
FAIL
FAIL    encoding/gob    35.425s
--- FAIL: TestFlagParser (0.04 seconds)
    fmt_test.go:778: Sprintf("%-1.2a", &flagprinter) => "[\xa0\x181.2a]", want "[%-1.2a]"
    fmt_test.go:778: Sprintf("%+1.2a", &flagprinter) => "[1\x001.2a]", want "[%+1.2a]"
    fmt_test.go:778: Sprintf("%-+1.2a", &flagprinter) => "[1\x00\x001.2a]", want "[%+-1.2a]"
    fmt_test.go:778: Sprintf("%-+1.2abc", &flagprinter) => "[\x90\x18@1.2a]bc", want "[%+-1.2a]bc"
    fmt_test.go:778: Sprintf("%-1.2abc", &flagprinter) => "[\xb0\x181.2a]bc", want "[%-1.2a]bc"
FAIL
FAIL    fmt 4.532s
--- FAIL: ExampleInspect (18.753686ms)
got:
2:92:9: p
3:73:7: c
3:113:11:   1.0
4:54:5: X
4:94:9: f
4:114:11:   3.14
4:174:17:   2
4:214:21:   c
want:
src.go:2:9: p
src.go:3:7: c
src.go:3:11:    1.0
src.go:4:5: X
src.go:4:9: f
src.go:4:11:    3.14
src.go:4:17:    2
src.go:4:21:    c
FAIL
FAIL    go/ast  1.131s
--- FAIL: TestFloatString (0.09 seconds)
    rat_test.go:159: #8 got 33.333 want -0.333
    rat_test.go:159: #9 got 66.6667 want -0.6667
    rat_test.go:159: #11 got .00 want 1.00
FAIL
FAIL    math/big    52.890s
--- FAIL: TestUnixConnLocalAndRemoteNames (0.02 seconds)
panic: runtime error: makeslice: cap out of range [recovered]
    panic: runtime error: makeslice: cap out of range
goroutine 354 [running]:
runtime.panic(0x24e380, 0x5106fc)
    /Users/rsc/g/go/src/pkg/runtime/panic.c:250 +0xb4
testing.func·005()
    /Users/rsc/g/go/src/pkg/testing/testing.go:392 +0x1a9
runtime.panic(0x24e380, 0x5106fc)
    /Users/rsc/g/go/src/pkg/runtime/panic.c:232 +0x110
strconv.quoteWith(0x0, 0xdeaddeaddeaddead, 0x22, 0x0, 0x0)
    /Users/rsc/g/go/src/pkg/strconv/quote.go:15 +0x8c
strconv.Quote(0x0, 0xdeaddeaddeaddead, 0x0, 0x0)
    /Users/rsc/g/go/src/pkg/strconv/quote.go:91 +0x5f
fmt.(*fmt).fmt_q(0xc208031990, 0x0, 0xdeaddeaddeaddead)
    /Users/rsc/g/go/src/pkg/fmt/format.go:334 +0x1a4
fmt.(*pp).fmtString(0xc208031930, 0x0, 0xdeaddeaddeaddead, 0x100000076)
    /Users/rsc/g/go/src/pkg/fmt/print.go:506 +0x116
fmt.(*pp).printReflectValue(0xc208031930, 0x1f7be0, 0xc208046760, 0x0, 0x186, ...)
    /Users/rsc/g/go/src/pkg/fmt/print.go:852 +0x4018
fmt.(*pp).printValue(0xc208031930, 0x1f7be0, 0xc208046760, 0x0, 0x186, ...)
    /Users/rsc/g/go/src/pkg/fmt/print.go:823 +0x54a
fmt.(*pp).printReflectValue(0xc208031930, 0x2416c0, 0xc208046760, 0x0, 0x196, ...)
    /Users/rsc/g/go/src/pkg/fmt/print.go:903 +0x46ce
fmt.(*pp).printValue(0xc208031930, 0x2416c0, 0xc208046760, 0x0, 0x196, ...)
    /Users/rsc/g/go/src/pkg/fmt/print.go:823 +0x54a
fmt.(*pp).printReflectValue(0xc208031930, 0x258fe0, 0xc208046760, 0x0, 0x160, ...)
    /Users/rsc/g/go/src/pkg/fmt/print.go:976 +0x3f74
fmt.(*pp).printArg(0xc208031930, 0x258fe0, 0xc208046760, 0x10000000076, 0x0, ...)
    /Users/rsc/g/go/src/pkg/fmt/print.go:785 +0x506
fmt.(*pp).doPrintf(0xc208031930, 0x299770, 0x15, 0x1adfe20, 0x2, ...)
    /Users/rsc/g/go/src/pkg/fmt/print.go:1150 +0x247d
fmt.Sprintf(0x299770, 0x15, 0x1adfe20, 0x2, 0x2, ...)
    /Users/rsc/g/go/src/pkg/fmt/print.go:203 +0x8e
testing.(*common).Fatalf(0xc2080741b0, 0x299770, 0x15, 0x1adfe20, 0x2, ...)
    /Users/rsc/g/go/src/pkg/testing/testing.go:320 +0x61
net.TestUnixConnLocalAndRemoteNames(0xc2080741b0)
    /Users/rsc/g/go/src/pkg/net/unix_test.go:202 +0xe3d
testing.tRunner(0xc2080741b0, 0x51b5d0)
    /Users/rsc/g/go/src/pkg/testing/testing.go:398 +0xbb
created by testing.RunTests
    /Users/rsc/g/go/src/pkg/testing/testing.go:479 +0x9e2
...
FAIL    net 18.086s
--- FAIL: TestAuthFailed (0.01 seconds)
    smtp_test.go:528: Auth: got error: 535 ase see www.example
        please see www.example.com, want: 535 Invalid credentials
        please see www.example.com
FAIL
FAIL    net/smtp    1.243s
--- FAIL: TestRFC959Lines (0.00 seconds)
    reader_test.go:244: #3: msg="\nC\x00\nC", want "A\nB\nC"
FAIL
FAIL    net/textproto   1.061s
--- FAIL: TestGenericTrieBuilding (0.05 seconds)
    replace_test.go:399: PrintTrie("abc;abdef;abdefgh;xx;xy;z")
        got
        pW=��ޭޭޭ�+..........gh+
        x��?�.y+
        z+
        want
        -
        a-
        .b-
        ..c+
        ..d-
        ...ef+
        .....gh+
        x-
        .x+
        .y+
        z+
    replace_test.go:399: PrintTrie("abracadabra;abracadabrakazam;abraham;abrasion")
        got
        .......��A��ޭޭޭ�..��<��ޭޭޭ�am+
        ....h.......am+
        ....s.......ion+
        want
        -
        a-
        .bra-
        ....c-
        .....adabra+
        ...........kazam+
        ....h-
        .....am+
        ....s-
        .....ion+
    replace_test.go:399: PrintTrie("aaa;aa;a;i;longerst;longer;long;xx;x;X;Y")
        got
        �ޭޭޭ�.a+
        i+
        l..............st+
        x..x+
        want
        -
        X+
        Y+
        a+
        .a+
        ..a+
        i+
        l-
        .ong+
        ....er+
        ......st+
        x+
        .x+
    replace_test.go:399: PrintTrie("foo;;foo;foo1")
        got
        .........1+
        want
        +
        f-
        .oo+
        ...1+
FAIL
FAIL    strings 1.526s
--- FAIL: TestParse (0.29 seconds)
    parse_test.go:289: pipeline with decl=("{{$x := .X|.Y}}"): got
            {{.Y}}
        expected
            {{$x := .X | .Y}}
    parse_test.go:289: nested pipeline=("{{.X (.Y .Z) (.A | .B .C) (.E)}}"): got
            {{��@��ޭޭޭ�.C) (.E)}}
        expected
            {{.X (.Y .Z) (.A | .B .C) (.E)}}
    parse_test.go:289: range 2 vars=("{{range $x, $y := .SI}}{{.}}{{end}}"): got
            {{range .SI}}{{.}}{{end}}
        expected
            {{range $x, $y := .SI}}{{.}}{{end}}
--- FAIL: TestParseCopy (0.36 seconds)
    parse_test.go:289: pipeline with decl=("{{$x := .X|.Y}}"): got
            {{.Y}}
        expected
            {{$x := .X | .Y}}
    parse_test.go:289: nested pipeline=("{{.X (.Y .Z) (.A | .B .C) (.E)}}"): got
            {{�ޭޭޭ�C.C) (.E)}}
        expected
            {{.X (.Y .Z) (.A | .B .C) (.E)}}
    parse_test.go:289: range 2 vars=("{{range $x, $y := .SI}}{{.}}{{end}}"): got
            {{range .SI}}{{.}}{{end}}
        expected
            {{range $x, $y := .SI}}{{.}}{{end}}
FAIL
FAIL    text/template/parse 1.603s

Owner changed to @rsc.

[rsc]

Comment 7:

Fixed.

Status changed to Fixed.