panic on Writer.Close, runtime error: index out of range

[toffaletti]

Unfortunately I don't have a test to reproduce this, but over hundreds of thousands of calls we started seeing a few of these panics:

runtime error: index out of range
goroutine 148112966 [running]:
net/http.func·011()
        /usr/lib/go/src/net/http/server.go:1130 +0xbb
github.com/klauspost/compress/flate.(*compressor).deflateNoSkip(0xc20e431600)
        _vendor/src/github.com/klauspost/compress/flate/deflate.go:594 +0xc68
github.com/klauspost/compress/flate.(*compressor).close(0xc20e431600, 0x0, 0x0)
        _vendor/src/github.com/klauspost/compress/flate/deflate.go:773 +0x49
github.com/klauspost/compress/flate.(*Writer).Close(0xc20e431600, 0x0, 0x0)
        _vendor/src/github.com/klauspost/compress/flate/deflate.go:854 

We never saw this before I updated our snapshot on Nov 2nd. The previous update was Sep 8th.

[klauspost]

Thanks for the report - I will investigate at once!

[klauspost]

Can you confirm that deflate.go:594 is also https://github.com/klauspost/compress/blob/master/flate/deflate.go#L594 ?

[toffaletti]

Yes, and unfortunately there are two indexes used on that line. I'm not sure which is out of range.

[shishkander]

@maruel has found this bug too https://code.google.com/p/chromium/issues/detail?id=552697#c4 .
The unittest in our code is a bit large but can reproduce this.

[klauspost]

From the other report it is clear that it is d.tokens.n that somehow gets too high. I am trying to recreate a scenario where that happens. Unfortunately my 500k files "webserver" test does not reproduce it.

I am of course continuing work on this.

[klauspost]

OK. Are you checking returned error values?

If you call Write after a write has already failed (in case of the chrominum bug) I can see this error occurring.

Close after a failed write will do the same.

Both are of course bugs, and will be fixed - I just want to be sure I haven't overlooked anything.

[maruel]

We reproduce in this case:

• Compress, writing to an io.Pipe().
• Using io.Copy() to write the input data to the compressor.
• Length of data is >= 64kb + 1.
• The read pipe from io.Pipe() is closed in the middle of the stream.

I was planning try to write a regression test before even bothering you about this but too late, eh. :)

[maruel]

@toffaletti (shameless) plug for panicparse to make your stack traces shorter.

[klauspost]

Thanks for helping identify this issue. I have merged the fix after testing.

[toffaletti]

In our code I don't control whether Write is called again after an error because our write is essentially json.NewEncoder(gzipWriter).Encode(). I will open a new issue if this fix didn't clear up the panic. Thanks so much for the prompt response despite so few details.