Skip to content

Commit c7c6a27

Browse files
klauspostklauspost
authored
Remove the last decoder allocation (#142)
* Remove the last decoder allocation ``` benchmark old ns/op new ns/op delta BenchmarkDecoder_DecodeAll/kppkn.gtb.zst-12 608499 610005 +0.25% BenchmarkDecoder_DecodeAll/geo.protodata.zst-12 136501 137699 +0.88% BenchmarkDecoder_DecodeAll/plrabn12.txt.zst-12 1967989 1944000 -1.22% BenchmarkDecoder_DecodeAll/lcet10.txt.zst-12 1448010 1444000 -0.28% BenchmarkDecoder_DecodeAll/asyoulik.txt.zst-12 504333 504333 +0.00% BenchmarkDecoder_DecodeAll/alice29.txt.zst-12 650494 652504 +0.31% BenchmarkDecoder_DecodeAll/html_x_4.zst-12 267000 267199 +0.07% BenchmarkDecoder_DecodeAll/paper-100k.pdf.zst-12 24840 24659 -0.73% BenchmarkDecoder_DecodeAll/fireworks.jpeg.zst-12 9594 9639 +0.47% BenchmarkDecoder_DecodeAll/urls.10K.zst-12 1662010 1670000 +0.48% BenchmarkDecoder_DecodeAll/html.zst-12 156400 155601 -0.51% benchmark old MB/s new MB/s speedup BenchmarkDecoder_DecodeAll/kppkn.gtb.zst-12 302.91 302.16 1.00x BenchmarkDecoder_DecodeAll/geo.protodata.zst-12 868.77 861.21 0.99x BenchmarkDecoder_DecodeAll/plrabn12.txt.zst-12 244.85 247.87 1.01x BenchmarkDecoder_DecodeAll/lcet10.txt.zst-12 294.72 295.54 1.00x BenchmarkDecoder_DecodeAll/asyoulik.txt.zst-12 248.21 248.21 1.00x BenchmarkDecoder_DecodeAll/alice29.txt.zst-12 233.81 233.09 1.00x BenchmarkDecoder_DecodeAll/html_x_4.zst-12 1534.08 1532.93 1.00x BenchmarkDecoder_DecodeAll/paper-100k.pdf.zst-12 4122.35 4152.47 1.01x BenchmarkDecoder_DecodeAll/fireworks.jpeg.zst-12 12828.87 12769.05 1.00x BenchmarkDecoder_DecodeAll/urls.10K.zst-12 422.43 420.41 1.00x BenchmarkDecoder_DecodeAll/html.zst-12 654.73 658.09 1.01x benchmark old allocs new allocs delta BenchmarkDecoder_DecodeAll/kppkn.gtb.zst-12 1 0 -100.00% BenchmarkDecoder_DecodeAll/geo.protodata.zst-12 1 0 -100.00% BenchmarkDecoder_DecodeAll/plrabn12.txt.zst-12 1 0 -100.00% BenchmarkDecoder_DecodeAll/lcet10.txt.zst-12 1 0 -100.00% BenchmarkDecoder_DecodeAll/asyoulik.txt.zst-12 1 0 -100.00% BenchmarkDecoder_DecodeAll/alice29.txt.zst-12 1 0 -100.00% BenchmarkDecoder_DecodeAll/html_x_4.zst-12 1 0 -100.00% BenchmarkDecoder_DecodeAll/paper-100k.pdf.zst-12 1 0 -100.00% BenchmarkDecoder_DecodeAll/fireworks.jpeg.zst-12 1 0 -100.00% BenchmarkDecoder_DecodeAll/urls.10K.zst-12 1 0 -100.00% BenchmarkDecoder_DecodeAll/html.zst-12 1 0 -100.00% benchmark old bytes new bytes delta BenchmarkDecoder_DecodeAll/kppkn.gtb.zst-12 32 0 -100.00% BenchmarkDecoder_DecodeAll/geo.protodata.zst-12 32 0 -100.00% BenchmarkDecoder_DecodeAll/plrabn12.txt.zst-12 32 0 -100.00% BenchmarkDecoder_DecodeAll/lcet10.txt.zst-12 32 0 -100.00% BenchmarkDecoder_DecodeAll/asyoulik.txt.zst-12 32 0 -100.00% BenchmarkDecoder_DecodeAll/alice29.txt.zst-12 32 0 -100.00% BenchmarkDecoder_DecodeAll/html_x_4.zst-12 32 0 -100.00% BenchmarkDecoder_DecodeAll/paper-100k.pdf.zst-12 32 0 -100.00% BenchmarkDecoder_DecodeAll/fireworks.jpeg.zst-12 32 0 -100.00% BenchmarkDecoder_DecodeAll/urls.10K.zst-12 32 0 -100.00% BenchmarkDecoder_DecodeAll/html.zst-12 32 0 -100.00% ```
1 parent 2325679 commit c7c6a27

File tree

3 files changed

+11
-6
lines changed

3 files changed

+11
-6
lines changed

‎zstd/README.md

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -34,7 +34,8 @@ For now, a high speed (fastest) and medium-fast (default) compressor has been im
3434
The "Fastest" compression ratio is roughly equivalent to zstd level 1.
3535
The "Default" compression ration is roughly equivalent to zstd level 3 (default).
3636

37-
In terms of speed, it is typically 2x as fast as the stdlib deflate/gzip in its fastest mode. The compression ratio compared to stdlib is around level 3, but usually 3x as fast.
37+
In terms of speed, it is typically 2x as fast as the stdlib deflate/gzip in its fastest mode.
38+
The compression ratio compared to stdlib is around level 3, but usually 3x as fast.
3839

3940
Compared to cgo zstd, the speed is around level 3 (default), but compression slightly worse, between level 1&2.
4041

@@ -217,7 +218,8 @@ silesia.tar zstd 3 211947520 66793301 1377 146.79
217218

218219
As part of the development process a *Snappy* -> *Zstandard* converter was also built.
219220

220-
This can convert a *framed* [Snappy Stream](https://godoc.org/github.com/golang/snappy#Writer) to a zstd stream. Note that a single block is not framed.
221+
This can convert a *framed* [Snappy Stream](https://godoc.org/github.com/golang/snappy#Writer) to a zstd stream.
222+
Note that a single block is not framed.
221223

222224
Conversion is done by converting the stream directly from Snappy without intermediate full decoding.
223225
Therefore the compression ratio is much less than what can be done by a full decompression

‎zstd/decoder.go

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -281,17 +281,17 @@ func (d *Decoder) DecodeAll(input, dst []byte) ([]byte, error) {
281281
}
282282
d.decoders <- block
283283
frame.rawInput = nil
284+
frame.bBuf = nil
284285
d.frames <- frame
285286
}()
287+
frame.bBuf = input
286288
if cap(dst) == 0 {
287289
// Allocate 1MB by default if nothing is provided.
288290
dst = make([]byte, 0, 1<<20)
289291
}
290292

291-
// Allocation here:
292-
br := byteBuf(input)
293293
for {
294-
err := frame.reset(&br)
294+
err := frame.reset(&frame.bBuf)
295295
if err == io.EOF {
296296
return dst, nil
297297
}
@@ -313,7 +313,7 @@ func (d *Decoder) DecodeAll(input, dst []byte) ([]byte, error) {
313313
if err != nil {
314314
return dst, err
315315
}
316-
if len(br) == 0 {
316+
if len(frame.bBuf) == 0 {
317317
break
318318
}
319319
}

‎zstd/framedec.go

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,9 @@ type frameDec struct {
3939

4040
rawInput byteBuffer
4141

42+
// Byte buffer that can be reused for small input blocks.
43+
bBuf byteBuf
44+
4245
// asyncRunning indicates whether the async routine processes input on 'decoding'.
4346
asyncRunning bool
4447
asyncRunningMu sync.Mutex

0 commit comments

Comments
 (0)