chunks: improve readability of compression benchmarks #7246
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dannykopping
approved these changes
Sep 26, 2022
For BenchmarkWrite: Use Go runtime facilities to report MB/s for each operation and compression ratio. Stop collecting all compressed chunks, as this blows up the memory of the benchmark and creates unrealistic test conditions for the later encoding. For BenchmarkRead: Replace reporting of MB/s which gave several values for each encoding as Go hunted for the right number of benchmark iterations. Add stats to the context for decoding, otherwise a new stats object is created each time round the loop. Re-order so all unsampled results come before all sampled results.
bboreham
force-pushed
the
compression-benchmarks
branch
from
6694b5f to
31f56cd
Compare
|
./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell. + ingester 0%
+ distributor 0%
+ querier 0%
+ querier/queryrange 0%
+ iter 0%
+ storage 0%
+ chunkenc 0%
+ logql 0%
+ loki 0% |
This was referenced Sep 27, 2022
owen-d
pushed a commit
that referenced
this pull request
Sep 27, 2022
**What this PR does / why we need it**: Bring in various improvements from upstream, particularly klauspost/compress#668 **Checklist** - [x] Reviewed the `CONTRIBUTING.md` guide - NA Documentation added - NA Tests updated - [x] `CHANGELOG.md` updated - NA Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` **Benchmarks** (using changes from #7246, ignoring lines that didn't change much) ``` name old time/op new time/op delta Write/ordered-flate-4 81.5ms ± 5% 78.7ms ± 2% -3.39% (p=0.016 n=5+5) Read/none_66_kB-4 4.16ms ± 6% 3.93ms ± 4% -5.51% (p=0.032 n=5+5) Read/flate_66_kB-4 168ms ± 4% 179ms ±11% +6.50% (p=0.032 n=5+5) Read/zstd_66_kB-4 334ms ±16% 152ms ± 3% -54.57% (p=0.008 n=5+5) Read/none_262_kB-4 3.93ms ± 1% 3.81ms ± 2% -2.91% (p=0.008 n=5+5) Read/zstd_262_kB-4 369ms ± 8% 131ms ± 1% -64.62% (p=0.008 n=5+5) Read/zstd_524_kB-4 466ms ±17% 146ms ± 2% -68.68% (p=0.008 n=5+5) Read/sample_zstd_66_kB-4 420ms ±18% 224ms ± 1% -46.69% (p=0.016 n=5+4) Read/sample_gzip_262_kB-4 214ms ± 1% 218ms ± 1% +1.90% (p=0.016 n=5+5) Read/sample_zstd_262_kB-4 418ms ±21% 200ms ± 2% -52.15% (p=0.016 n=5+4) Read/sample_zstd_524_kB-4 576ms ±35% 234ms ± 1% -59.42% (p=0.008 n=5+5) name old speed new speed delta Write/ordered-gzip-4 267MB/s ±10% 284MB/s ± 2% +6.51% (p=0.008 n=5+5) Write/ordered-flate-4 274MB/s ± 4% 291MB/s ± 2% +6.49% (p=0.008 n=5+5) Read/none_66_kB-4 1.82GB/s ± 6% 1.92GB/s ± 4% +5.78% (p=0.032 n=5+5) Read/gzip_66_kB-4 669MB/s ± 4% 698MB/s ± 1% +4.30% (p=0.008 n=5+5) Read/zstd_66_kB-4 425MB/s ±14% 930MB/s ± 3% +119.12% (p=0.008 n=5+5) Read/none_262_kB-4 1.93GB/s ± 1% 1.99GB/s ± 2% +3.00% (p=0.008 n=5+5) Read/gzip_262_kB-4 701MB/s ± 5% 721MB/s ± 2% +2.96% (p=0.032 n=5+5) Read/zstd_262_kB-4 357MB/s ± 8% 1009MB/s ± 1% +182.89% (p=0.008 n=5+5) Read/flate_524_kB-4 750MB/s ± 1% 763MB/s ± 1% +1.74% (p=0.008 n=5+5) Read/zstd_524_kB-4 365MB/s ±18% 1149MB/s ± 2% +215.34% (p=0.008 n=5+5) Read/sample_zstd_66_kB-4 341MB/s ±17% 616MB/s ± 9% +80.67% (p=0.008 n=5+5) Read/sample_zstd_262_kB-4 318MB/s ±18% 658MB/s ± 2% +107.08% (p=0.016 n=5+4) Read/sample_gzip_524_kB-4 521MB/s ± 1% 530MB/s ± 1% +1.72% (p=0.032 n=5+5) Read/sample_zstd_524_kB-4 299MB/s ±28% 718MB/s ± 1% +140.23% (p=0.008 n=5+5) name old %compressed new %compressed delta Write/ordered-gzip-4 6.89 ± 0% 6.73 ± 0% -2.32% (p=0.000 n=5+4) Write/ordered-flate-4 6.89 ± 0% 6.69 ± 0% -2.84% (p=0.008 n=5+5) Write/ordered-zstd-4 6.17 ± 0% 6.16 ± 0% -0.02% (p=0.008 n=5+5) Write/unordered-gzip-4 6.89 ± 0% 6.73 ± 0% -2.32% (p=0.008 n=5+5) Write/unordered-flate-4 6.89 ± 0% 6.70 ± 0% -2.81% (p=0.000 n=4+5) Write/unordered-zstd-4 6.17 ± 0% 6.16 ± 0% -0.02% (p=0.008 n=5+5) name old alloc/op new alloc/op delta Write/unordered-lz4-4 18.8MB ± 4% 17.8MB ± 1% -5.28% (p=0.008 n=5+5) Read/gzip_66_kB-4 713kB ± 0% 735kB ± 1% +2.99% (p=0.008 n=5+5) Read/flate_66_kB-4 715kB ± 0% 733kB ± 0% +2.57% (p=0.008 n=5+5) Read/zstd_66_kB-4 423MB ± 0% 1MB ± 0% -99.65% (p=0.016 n=5+4) Read/gzip_262_kB-4 198kB ± 0% 204kB ± 0% +3.07% (p=0.016 n=4+5) Read/lz4-64k_262_kB-4 169kB ± 0% 169kB ± 0% -0.07% (p=0.000 n=5+4) Read/flate_262_kB-4 197kB ± 0% 205kB ± 0% +3.77% (p=0.008 n=5+5) Read/zstd_262_kB-4 553MB ± 0% 0MB ± 0% -99.92% (p=0.016 n=5+4) Read/gzip_524_kB-4 96.2kB ± 0% 98.8kB ± 0% +2.71% (p=0.016 n=4+5) Read/flate_524_kB-4 95.4kB ± 0% 98.7kB ± 0% +3.46% (p=0.008 n=5+5) Read/zstd_524_kB-4 786MB ± 0% 1MB ± 6% -99.93% (p=0.008 n=5+5) Read/sample_gzip_66_kB-4 686kB ± 0% 703kB ± 0% +2.48% (p=0.029 n=4+4) Read/sample_flate_66_kB-4 688kB ± 0% 710kB ± 1% +3.08% (p=0.016 n=4+5) Read/sample_zstd_66_kB-4 423MB ± 0% 2MB ± 0% -99.59% (p=0.016 n=5+4) Read/sample_gzip_262_kB-4 194kB ± 0% 200kB ± 0% +3.10% (p=0.016 n=4+5) Read/sample_flate_262_kB-4 194kB ± 0% 201kB ± 0% +3.77% (p=0.016 n=4+5) Read/sample_zstd_262_kB-4 553MB ± 0% 0MB ± 0% -99.91% (p=0.016 n=5+4) Read/sample_gzip_524_kB-4 95.4kB ± 0% 97.9kB ± 0% +2.65% (p=0.008 n=5+5) Read/sample_flate_524_kB-4 95.2kB ± 0% 98.8kB ± 0% +3.71% (p=0.016 n=5+4) Read/sample_zstd_524_kB-4 786MB ± 0% 1MB ± 0% -99.90% (p=0.008 n=5+5) name old allocs/op new allocs/op delta Write/ordered-gzip-4 701 ± 0% 718 ± 1% +2.40% (p=0.008 n=5+5) Write/ordered-lz4-256k-4 656 ± 0% 657 ± 0% +0.21% (p=0.048 n=5+5) Write/ordered-flate-4 626 ± 0% 634 ± 1% +1.37% (p=0.008 n=5+5) Write/ordered-zstd-4 2.07k ± 0% 1.44k ± 0% -30.49% (p=0.008 n=5+5) Write/unordered-gzip-4 418k ± 0% 428k ± 0% +2.39% (p=0.008 n=5+5) Write/unordered-flate-4 418k ± 0% 430k ± 0% +2.89% (p=0.008 n=5+5) Write/unordered-zstd-4 469k ± 0% 468k ± 0% -0.12% (p=0.008 n=5+5) Read/gzip_66_kB-4 13.2k ± 0% 13.5k ± 0% +2.54% (p=0.016 n=5+4) Read/lz4-256k_66_kB-4 13.7k ± 0% 13.7k ± 0% -0.01% (p=0.029 n=4+4) Read/flate_66_kB-4 13.2k ± 0% 13.6k ± 0% +2.72% (p=0.008 n=5+5) Read/zstd_66_kB-4 18.0k ± 1% 10.8k ± 0% -40.19% (p=0.008 n=5+5) Read/gzip_262_kB-4 4.61k ± 0% 4.76k ± 0% +3.17% (p=0.016 n=4+5) Read/flate_262_kB-4 4.61k ± 0% 4.78k ± 0% +3.62% (p=0.008 n=5+5) Read/zstd_262_kB-4 7.10k ± 2% 2.54k ± 0% -64.22% (p=0.016 n=5+4) Read/gzip_524_kB-4 2.62k ± 0% 2.69k ± 0% +2.36% (p=0.016 n=4+5) Read/flate_524_kB-4 2.62k ± 0% 2.70k ± 0% +3.09% (p=0.008 n=5+5) Read/zstd_524_kB-4 6.13k ± 2% 1.63k ± 0% -73.40% (p=0.008 n=5+5) Read/sample_gzip_66_kB-4 13.2k ± 0% 13.5k ± 0% +2.54% (p=0.029 n=4+4) Read/sample_flate_66_kB-4 13.2k ± 0% 13.6k ± 0% +2.88% (p=0.016 n=4+5) Read/sample_zstd_66_kB-4 18.0k ± 1% 10.8k ± 0% -40.00% (p=0.008 n=5+5) Read/sample_gzip_262_kB-4 4.63k ± 0% 4.77k ± 0% +3.16% (p=0.016 n=4+5) Read/sample_flate_262_kB-4 4.62k ± 0% 4.79k ± 0% +3.61% (p=0.016 n=4+5) Read/sample_zstd_262_kB-4 7.12k ± 1% 2.54k ± 0% -64.28% (p=0.008 n=5+5) Read/sample_gzip_524_kB-4 2.63k ± 0% 2.69k ± 0% +2.36% (p=0.008 n=5+5) Read/sample_flate_524_kB-4 2.63k ± 0% 2.72k ± 0% +3.31% (p=0.016 n=5+4) Read/sample_zstd_524_kB-4 6.11k ± 2% 1.64k ± 0% -73.23% (p=0.008 n=5+5) ```
owen-d
added a commit
that referenced
this pull request
Sep 30, 2022
…me corrupt inputs (#7264) **What this PR does / why we need it**: This is much faster, since we don't pay for byte-at-a-time reads from the buffer. For best performance with gzip and lz4, we still want a bufio.Reader. We don't need the pool of buffered readers now. Other compression formats have their own internal buffers so don't need extra buffering. **Checklist** - [x] Reviewed the `CONTRIBUTING.md` guide - NA Documentation added - NA Tests updated - [x] `CHANGELOG.md` updated - NA Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` (using changes from #7246, ignoring lines that didn't change much) ``` name old time/op new time/op delta Write/ordered-lz4-256k-4 24.1ms ± 7% 22.1ms ± 2% -8.52% (p=0.008 n=5+5) Write/unordered-lz4-4 54.6ms ±29% 49.1ms ± 1% -10.04% (p=0.016 n=5+4) Read/none_66_kB-4 4.16ms ± 6% 2.54ms ± 2% -38.87% (p=0.008 n=5+5) Read/gzip_66_kB-4 177ms ± 4% 161ms ± 4% -9.09% (p=0.008 n=5+5) Read/lz4-64k_66_kB-4 59.5ms ± 2% 49.5ms ± 3% -16.80% (p=0.008 n=5+5) Read/lz4-256k_66_kB-4 62.9ms ± 0% 51.2ms ± 1% -18.58% (p=0.016 n=4+5) Read/lz4-1M_66_kB-4 62.9ms ± 2% 51.5ms ± 1% -18.03% (p=0.008 n=5+5) Read/lz4_66_kB-4 63.7ms ± 4% 51.2ms ± 1% -19.68% (p=0.008 n=5+5) Read/snappy_66_kB-4 51.1ms ± 2% 41.9ms ± 1% -17.95% (p=0.016 n=5+4) Read/flate_66_kB-4 168ms ± 4% 150ms ± 2% -10.49% (p=0.008 n=5+5) Read/zstd_66_kB-4 334ms ±16% 272ms ± 6% -18.52% (p=0.008 n=5+5) Read/none_262_kB-4 3.93ms ± 1% 2.48ms ± 2% -36.91% (p=0.008 n=5+5) Read/gzip_262_kB-4 159ms ± 5% 143ms ± 2% -10.12% (p=0.008 n=5+5) Read/lz4-64k_262_kB-4 53.5ms ± 1% 43.2ms ± 3% -19.35% (p=0.008 n=5+5) Read/lz4-256k_262_kB-4 55.8ms ± 1% 45.3ms ± 2% -18.75% (p=0.008 n=5+5) Read/lz4-1M_262_kB-4 57.5ms ± 2% 45.5ms ± 2% -20.91% (p=0.008 n=5+5) Read/lz4_262_kB-4 56.8ms ± 2% 45.3ms ± 2% -20.26% (p=0.008 n=5+5) Read/snappy_262_kB-4 46.0ms ± 2% 37.5ms ± 1% -18.57% (p=0.008 n=5+5) Read/flate_262_kB-4 153ms ± 3% 134ms ± 2% -12.33% (p=0.008 n=5+5) Read/none_524_kB-4 3.85ms ± 2% 2.46ms ± 2% -36.04% (p=0.008 n=5+5) Read/gzip_524_kB-4 128ms ± 2% 119ms ± 2% -7.64% (p=0.008 n=5+5) Read/lz4-64k_524_kB-4 44.6ms ± 3% 34.7ms ± 1% -22.19% (p=0.008 n=5+5) Read/lz4-256k_524_kB-4 45.4ms ± 1% 36.7ms ± 5% -19.07% (p=0.008 n=5+5) Read/lz4-1M_524_kB-4 47.3ms ± 1% 38.6ms ± 2% -18.36% (p=0.008 n=5+5) Read/lz4_524_kB-4 47.4ms ± 1% 38.0ms ± 1% -19.70% (p=0.008 n=5+5) Read/snappy_524_kB-4 37.3ms ± 1% 31.1ms ± 3% -16.47% (p=0.008 n=5+5) Read/flate_524_kB-4 122ms ± 1% 111ms ± 3% -9.04% (p=0.008 n=5+5) Read/sample_none_66_kB-4 7.81ms ± 1% 6.62ms ± 6% -15.24% (p=0.008 n=5+5) Read/sample_gzip_66_kB-4 237ms ± 2% 219ms ± 1% -7.57% (p=0.008 n=5+5) Read/sample_lz4-64k_66_kB-4 105ms ± 2% 92ms ± 1% -12.03% (p=0.008 n=5+5) Read/sample_lz4-256k_66_kB-4 109ms ± 1% 98ms ± 2% -9.65% (p=0.008 n=5+5) Read/sample_lz4-1M_66_kB-4 113ms ± 8% 98ms ± 2% -13.66% (p=0.008 n=5+5) Read/sample_lz4_66_kB-4 109ms ± 1% 99ms ± 2% -9.29% (p=0.008 n=5+5) Read/sample_snappy_66_kB-4 86.6ms ± 1% 77.2ms ± 6% -10.92% (p=0.008 n=5+5) Read/sample_flate_66_kB-4 236ms ± 8% 214ms ± 3% -9.11% (p=0.008 n=5+5) Read/sample_none_262_kB-4 7.70ms ± 2% 6.34ms ± 2% -17.67% (p=0.008 n=5+5) Read/sample_lz4-64k_262_kB-4 94.7ms ± 1% 84.4ms ± 2% -10.90% (p=0.008 n=5+5) Read/sample_lz4-256k_262_kB-4 98.4ms ± 1% 87.7ms ± 2% -10.93% (p=0.008 n=5+5) Read/sample_lz4-1M_262_kB-4 101ms ± 2% 90ms ± 2% -11.19% (p=0.008 n=5+5) Read/sample_lz4_262_kB-4 100ms ± 2% 90ms ± 3% -10.78% (p=0.008 n=5+5) Read/sample_snappy_262_kB-4 77.9ms ± 2% 68.7ms ± 2% -11.84% (p=0.008 n=5+5) Read/sample_flate_262_kB-4 211ms ± 2% 190ms ± 2% -9.86% (p=0.008 n=5+5) Read/sample_none_524_kB-4 7.82ms ± 2% 6.43ms ± 2% -17.80% (p=0.008 n=5+5) Read/sample_gzip_524_kB-4 176ms ± 1% 164ms ± 2% -6.80% (p=0.008 n=5+5) Read/sample_lz4-64k_524_kB-4 77.7ms ± 1% 69.8ms ± 4% -10.10% (p=0.008 n=5+5) Read/sample_lz4-256k_524_kB-4 81.7ms ± 2% 72.1ms ± 3% -11.79% (p=0.008 n=5+5) Read/sample_lz4-1M_524_kB-4 83.8ms ± 1% 74.4ms ± 1% -11.21% (p=0.008 n=5+5) Read/sample_lz4_524_kB-4 83.2ms ± 0% 76.1ms ± 8% -8.53% (p=0.016 n=4+5) Read/sample_snappy_524_kB-4 63.6ms ± 2% 56.5ms ± 2% -11.23% (p=0.008 n=5+5) Read/sample_flate_524_kB-4 172ms ± 2% 157ms ± 1% -8.88% (p=0.008 n=5+5) name old speed new speed delta Write/ordered-lz4-256k-4 687MB/s ± 6% 749MB/s ± 2% +9.09% (p=0.008 n=5+5) Write/unordered-lz4-4 313MB/s ±24% 341MB/s ± 1% +9.23% (p=0.016 n=5+4) Read/none_66_kB-4 1.82GB/s ± 6% 2.97GB/s ± 2% +63.42% (p=0.008 n=5+5) Read/gzip_66_kB-4 669MB/s ± 4% 736MB/s ± 4% +10.00% (p=0.008 n=5+5) Read/lz4-64k_66_kB-4 1.42GB/s ± 2% 1.71GB/s ± 3% +20.21% (p=0.008 n=5+5) Read/lz4-256k_66_kB-4 1.43GB/s ± 0% 1.76GB/s ± 1% +22.83% (p=0.016 n=4+5) Read/lz4-1M_66_kB-4 1.43GB/s ± 2% 1.74GB/s ± 1% +21.99% (p=0.008 n=5+5) Read/lz4_66_kB-4 1.41GB/s ± 4% 1.76GB/s ± 1% +24.44% (p=0.008 n=5+5) Read/snappy_66_kB-4 1.29GB/s ± 2% 1.57GB/s ± 1% +21.86% (p=0.016 n=5+4) Read/flate_66_kB-4 710MB/s ± 4% 793MB/s ± 2% +11.70% (p=0.008 n=5+5) Read/zstd_66_kB-4 425MB/s ±14% 517MB/s ± 6% +21.74% (p=0.008 n=5+5) Read/none_262_kB-4 1.93GB/s ± 1% 3.06GB/s ± 2% +58.54% (p=0.008 n=5+5) Read/gzip_262_kB-4 701MB/s ± 5% 779MB/s ± 2% +11.20% (p=0.008 n=5+5) Read/lz4-64k_262_kB-4 1.47GB/s ± 1% 1.82GB/s ± 3% +24.03% (p=0.008 n=5+5) Read/lz4-256k_262_kB-4 1.48GB/s ± 1% 1.82GB/s ± 2% +23.09% (p=0.008 n=5+5) Read/lz4-1M_262_kB-4 1.46GB/s ± 2% 1.84GB/s ± 2% +26.44% (p=0.008 n=5+5) Read/lz4_262_kB-4 1.48GB/s ± 2% 1.85GB/s ± 2% +25.42% (p=0.008 n=5+5) Read/snappy_262_kB-4 1.31GB/s ± 2% 1.61GB/s ± 1% +22.80% (p=0.008 n=5+5) Read/flate_262_kB-4 729MB/s ± 3% 831MB/s ± 2% +14.05% (p=0.008 n=5+5) Read/none_524_kB-4 1.97GB/s ± 2% 3.08GB/s ± 2% +56.33% (p=0.008 n=5+5) Read/gzip_524_kB-4 714MB/s ± 2% 773MB/s ± 2% +8.27% (p=0.008 n=5+5) Read/lz4-64k_524_kB-4 1.47GB/s ± 3% 1.88GB/s ± 1% +28.48% (p=0.008 n=5+5) Read/lz4-256k_524_kB-4 1.50GB/s ± 1% 1.86GB/s ± 5% +23.63% (p=0.008 n=5+5) Read/lz4-1M_524_kB-4 1.49GB/s ± 1% 1.83GB/s ± 2% +22.51% (p=0.008 n=5+5) Read/lz4_524_kB-4 1.49GB/s ± 1% 1.86GB/s ± 1% +24.53% (p=0.008 n=5+5) Read/snappy_524_kB-4 1.34GB/s ± 1% 1.60GB/s ± 3% +19.75% (p=0.008 n=5+5) Read/flate_524_kB-4 750MB/s ± 1% 824MB/s ± 3% +9.97% (p=0.008 n=5+5) Read/sample_none_66_kB-4 967MB/s ± 1% 1142MB/s ± 6% +18.13% (p=0.008 n=5+5) Read/sample_gzip_66_kB-4 501MB/s ± 2% 542MB/s ± 1% +8.19% (p=0.008 n=5+5) Read/sample_lz4-64k_66_kB-4 811MB/s ± 2% 922MB/s ± 1% +13.66% (p=0.008 n=5+5) Read/sample_lz4-256k_66_kB-4 825MB/s ± 1% 913MB/s ± 2% +10.68% (p=0.008 n=5+5) Read/sample_lz4-1M_66_kB-4 795MB/s ± 7% 920MB/s ± 1% +15.67% (p=0.008 n=5+5) Read/sample_lz4_66_kB-4 824MB/s ± 1% 909MB/s ± 2% +10.25% (p=0.008 n=5+5) Read/sample_snappy_66_kB-4 759MB/s ± 1% 852MB/s ± 6% +12.37% (p=0.008 n=5+5) Read/sample_flate_66_kB-4 506MB/s ± 7% 556MB/s ± 4% +9.88% (p=0.008 n=5+5) Read/sample_none_262_kB-4 983MB/s ± 2% 1194MB/s ± 2% +21.47% (p=0.008 n=5+5) Read/sample_lz4-64k_262_kB-4 830MB/s ± 1% 932MB/s ± 2% +12.24% (p=0.008 n=5+5) Read/sample_lz4-256k_262_kB-4 839MB/s ± 1% 942MB/s ± 2% +12.28% (p=0.008 n=5+5) Read/sample_lz4-1M_262_kB-4 829MB/s ± 2% 934MB/s ± 2% +12.60% (p=0.008 n=5+5) Read/sample_lz4_262_kB-4 836MB/s ± 2% 937MB/s ± 3% +12.11% (p=0.008 n=5+5) Read/sample_snappy_262_kB-4 774MB/s ± 2% 878MB/s ± 2% +13.44% (p=0.008 n=5+5) Read/sample_flate_262_kB-4 529MB/s ± 2% 587MB/s ± 2% +10.94% (p=0.008 n=5+5) Read/sample_none_524_kB-4 970MB/s ± 2% 1180MB/s ± 2% +21.64% (p=0.008 n=5+5) Read/sample_gzip_524_kB-4 521MB/s ± 1% 559MB/s ± 2% +7.31% (p=0.008 n=5+5) Read/sample_lz4-64k_524_kB-4 842MB/s ± 1% 937MB/s ± 4% +11.30% (p=0.008 n=5+5) Read/sample_lz4-256k_524_kB-4 834MB/s ± 2% 945MB/s ± 3% +13.37% (p=0.008 n=5+5) Read/sample_lz4-1M_524_kB-4 842MB/s ± 1% 949MB/s ± 1% +12.63% (p=0.008 n=5+5) Read/sample_lz4_524_kB-4 849MB/s ± 0% 929MB/s ± 7% +9.53% (p=0.016 n=4+5) Read/sample_snappy_524_kB-4 783MB/s ± 2% 882MB/s ± 2% +12.65% (p=0.008 n=5+5) Read/sample_flate_524_kB-4 534MB/s ± 2% 585MB/s ± 1% +9.73% (p=0.008 n=5+5) name old alloc/op new alloc/op delta Write/unordered-lz4-4 18.8MB ± 4% 17.2MB ± 7% -8.28% (p=0.008 n=5+5) Read/none_66_kB-4 41.9kB ± 0% 43.7kB ± 0% +4.41% (p=0.008 n=5+5) Read/gzip_66_kB-4 713kB ± 0% 741kB ± 0% +3.85% (p=0.016 n=5+4) Read/lz4-64k_66_kB-4 730kB ± 1% 748kB ± 0% +2.53% (p=0.016 n=5+4) Read/lz4-256k_66_kB-4 770kB ± 0% 806kB ± 3% +4.70% (p=0.016 n=4+5) Read/lz4-1M_66_kB-4 770kB ± 0% 812kB ± 4% +5.43% (p=0.016 n=4+5) Read/lz4_66_kB-4 770kB ± 0% 792kB ± 0% +2.83% (p=0.029 n=4+4) Read/snappy_66_kB-4 361kB ± 0% 376kB ± 0% +4.02% (p=0.008 n=5+5) Read/flate_66_kB-4 715kB ± 0% 743kB ± 0% +3.88% (p=0.016 n=5+4) Read/none_262_kB-4 11.2kB ± 0% 11.7kB ± 0% +4.12% (p=0.016 n=5+4) Read/gzip_262_kB-4 198kB ± 0% 203kB ± 0% +2.93% (p=0.016 n=4+5) Read/lz4-64k_262_kB-4 169kB ± 0% 174kB ± 0% +2.75% (p=0.008 n=5+5) Read/lz4-256k_262_kB-4 177kB ± 0% 182kB ± 0% +2.79% (p=0.008 n=5+5) Read/lz4-1M_262_kB-4 181kB ± 0% 186kB ± 0% +2.77% (p=0.008 n=5+5) Read/lz4_262_kB-4 181kB ± 0% 186kB ± 0% +2.73% (p=0.016 n=5+4) Read/snappy_262_kB-4 87.8kB ± 0% 90.1kB ± 0% +2.58% (p=0.008 n=5+5) Read/flate_262_kB-4 197kB ± 0% 203kB ± 0% +2.63% (p=0.016 n=5+4) Read/zstd_262_kB-4 553MB ± 0% 552MB ± 0% -0.15% (p=0.032 n=5+5) Read/none_524_kB-4 5.94kB ± 0% 6.16kB ± 0% +3.76% (p=0.016 n=5+4) Read/gzip_524_kB-4 96.2kB ± 0% 98.3kB ± 0% +2.16% (p=0.016 n=4+5) Read/lz4-64k_524_kB-4 70.9kB ± 0% 72.8kB ± 0% +2.71% (p=0.016 n=4+5) Read/lz4-256k_524_kB-4 73.8kB ± 0% 75.8kB ± 0% +2.68% (p=0.008 n=5+5) Read/lz4-1M_524_kB-4 76.5kB ± 0% 78.7kB ± 0% +2.77% (p=0.016 n=4+5) Read/lz4_524_kB-4 76.5kB ± 0% 78.6kB ± 0% +2.74% (p=0.016 n=5+4) Read/snappy_524_kB-4 39.1kB ± 0% 39.5kB ± 0% +1.19% (p=0.008 n=5+5) Read/flate_524_kB-4 95.4kB ± 0% 97.3kB ± 0% +2.06% (p=0.008 n=5+5) Read/sample_none_66_kB-4 39.9kB ± 0% 41.7kB ± 0% +4.62% (p=0.008 n=5+5) Read/sample_gzip_66_kB-4 686kB ± 0% 715kB ± 0% +4.23% (p=0.016 n=4+5) Read/sample_lz4-64k_66_kB-4 707kB ± 0% 728kB ± 0% +2.88% (p=0.016 n=5+4) Read/sample_lz4-256k_66_kB-4 749kB ± 0% 783kB ± 4% +4.65% (p=0.008 n=5+5) Read/sample_lz4_66_kB-4 748kB ± 0% 770kB ± 0% +2.91% (p=0.016 n=5+4) Read/sample_snappy_66_kB-4 350kB ± 0% 364kB ± 0% +4.04% (p=0.029 n=4+4) Read/sample_flate_66_kB-4 688kB ± 0% 716kB ± 0% +4.08% (p=0.029 n=4+4) Read/sample_none_262_kB-4 10.6kB ± 0% 11.0kB ± 0% +4.23% (p=0.016 n=5+4) Read/sample_gzip_262_kB-4 194kB ± 0% 199kB ± 0% +2.48% (p=0.016 n=4+5) Read/sample_lz4-64k_262_kB-4 165kB ± 0% 169kB ± 0% +2.86% (p=0.016 n=4+5) Read/sample_lz4-256k_262_kB-4 172kB ± 0% 177kB ± 0% +2.92% (p=0.016 n=4+5) Read/sample_lz4-1M_262_kB-4 176kB ± 0% 181kB ± 0% +2.91% (p=0.029 n=4+4) Read/sample_lz4_262_kB-4 176kB ± 0% 181kB ± 0% +2.81% (p=0.016 n=5+4) Read/sample_snappy_262_kB-4 88.1kB ± 0% 90.8kB ± 0% +3.11% (p=0.029 n=4+4) Read/sample_flate_262_kB-4 194kB ± 0% 198kB ± 0% +2.13% (p=0.029 n=4+4) Read/sample_none_524_kB-4 5.56kB ± 0% 5.76kB ± 0% +3.65% (p=0.016 n=5+4) Read/sample_gzip_524_kB-4 95.4kB ± 0% 96.9kB ± 0% +1.56% (p=0.008 n=5+5) Read/sample_lz4-64k_524_kB-4 69.0kB ± 0% 71.0kB ± 0% +2.87% (p=0.008 n=5+5) Read/sample_lz4-256k_524_kB-4 71.8kB ± 0% 73.8kB ± 0% +2.79% (p=0.016 n=4+5) Read/sample_lz4-1M_524_kB-4 74.5kB ± 0% 76.6kB ± 0% +2.75% (p=0.008 n=5+5) Read/sample_lz4_524_kB-4 74.5kB ± 0% 188.4kB ±89% +152.98% (p=0.008 n=5+5) Read/sample_snappy_524_kB-4 40.7kB ± 1% 41.3kB ± 1% +1.58% (p=0.008 n=5+5) Read/sample_flate_524_kB-4 95.2kB ± 0% 96.2kB ± 0% +0.95% (p=0.008 n=5+5) name old allocs/op new allocs/op delta Read/gzip_66_kB-4 13.2k ± 0% 13.2k ± 0% -0.04% (p=0.000 n=5+4) Read/zstd_66_kB-4 18.0k ± 1% 17.8k ± 1% -1.06% (p=0.032 n=5+5) Read/gzip_262_kB-4 4.61k ± 0% 4.61k ± 0% -0.11% (p=0.029 n=4+4) Read/flate_262_kB-4 4.61k ± 0% 4.60k ± 0% -0.11% (p=0.008 n=5+5) Read/zstd_262_kB-4 7.10k ± 2% 6.88k ± 3% -3.01% (p=0.008 n=5+5) Read/gzip_524_kB-4 2.62k ± 0% 2.62k ± 0% -0.06% (p=0.000 n=4+5) Read/zstd_524_kB-4 6.13k ± 2% 5.94k ± 2% -3.03% (p=0.024 n=5+5) Read/sample_flate_66_kB-4 13.2k ± 0% 13.2k ± 0% -0.01% (p=0.029 n=4+4) Read/sample_flate_262_kB-4 4.62k ± 0% 4.61k ± 0% -0.22% (p=0.029 n=4+4) Read/sample_zstd_262_kB-4 7.12k ± 1% 6.97k ± 1% -2.11% (p=0.008 n=5+5) Read/sample_gzip_524_kB-4 2.63k ± 0% 2.63k ± 0% -0.11% (p=0.008 n=5+5) Read/sample_flate_524_kB-4 2.63k ± 0% 2.62k ± 0% -0.15% (p=0.008 n=5+5) Read/sample_zstd_524_kB-4 6.11k ± 2% 5.89k ± 1% -3.68% (p=0.016 n=5+4) ``` Co-authored-by: Owen Diehl <[email protected]>
kavirajk
approved these changes
Oct 11, 2022
lxwzy
pushed a commit
to lxwzy/loki
that referenced
this pull request
Nov 7, 2022
**What this PR does / why we need it**: Bring in various improvements from upstream, particularly klauspost/compress#668 **Checklist** - [x] Reviewed the `CONTRIBUTING.md` guide - NA Documentation added - NA Tests updated - [x] `CHANGELOG.md` updated - NA Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` **Benchmarks** (using changes from grafana#7246, ignoring lines that didn't change much) ``` name old time/op new time/op delta Write/ordered-flate-4 81.5ms ± 5% 78.7ms ± 2% -3.39% (p=0.016 n=5+5) Read/none_66_kB-4 4.16ms ± 6% 3.93ms ± 4% -5.51% (p=0.032 n=5+5) Read/flate_66_kB-4 168ms ± 4% 179ms ±11% +6.50% (p=0.032 n=5+5) Read/zstd_66_kB-4 334ms ±16% 152ms ± 3% -54.57% (p=0.008 n=5+5) Read/none_262_kB-4 3.93ms ± 1% 3.81ms ± 2% -2.91% (p=0.008 n=5+5) Read/zstd_262_kB-4 369ms ± 8% 131ms ± 1% -64.62% (p=0.008 n=5+5) Read/zstd_524_kB-4 466ms ±17% 146ms ± 2% -68.68% (p=0.008 n=5+5) Read/sample_zstd_66_kB-4 420ms ±18% 224ms ± 1% -46.69% (p=0.016 n=5+4) Read/sample_gzip_262_kB-4 214ms ± 1% 218ms ± 1% +1.90% (p=0.016 n=5+5) Read/sample_zstd_262_kB-4 418ms ±21% 200ms ± 2% -52.15% (p=0.016 n=5+4) Read/sample_zstd_524_kB-4 576ms ±35% 234ms ± 1% -59.42% (p=0.008 n=5+5) name old speed new speed delta Write/ordered-gzip-4 267MB/s ±10% 284MB/s ± 2% +6.51% (p=0.008 n=5+5) Write/ordered-flate-4 274MB/s ± 4% 291MB/s ± 2% +6.49% (p=0.008 n=5+5) Read/none_66_kB-4 1.82GB/s ± 6% 1.92GB/s ± 4% +5.78% (p=0.032 n=5+5) Read/gzip_66_kB-4 669MB/s ± 4% 698MB/s ± 1% +4.30% (p=0.008 n=5+5) Read/zstd_66_kB-4 425MB/s ±14% 930MB/s ± 3% +119.12% (p=0.008 n=5+5) Read/none_262_kB-4 1.93GB/s ± 1% 1.99GB/s ± 2% +3.00% (p=0.008 n=5+5) Read/gzip_262_kB-4 701MB/s ± 5% 721MB/s ± 2% +2.96% (p=0.032 n=5+5) Read/zstd_262_kB-4 357MB/s ± 8% 1009MB/s ± 1% +182.89% (p=0.008 n=5+5) Read/flate_524_kB-4 750MB/s ± 1% 763MB/s ± 1% +1.74% (p=0.008 n=5+5) Read/zstd_524_kB-4 365MB/s ±18% 1149MB/s ± 2% +215.34% (p=0.008 n=5+5) Read/sample_zstd_66_kB-4 341MB/s ±17% 616MB/s ± 9% +80.67% (p=0.008 n=5+5) Read/sample_zstd_262_kB-4 318MB/s ±18% 658MB/s ± 2% +107.08% (p=0.016 n=5+4) Read/sample_gzip_524_kB-4 521MB/s ± 1% 530MB/s ± 1% +1.72% (p=0.032 n=5+5) Read/sample_zstd_524_kB-4 299MB/s ±28% 718MB/s ± 1% +140.23% (p=0.008 n=5+5) name old %compressed new %compressed delta Write/ordered-gzip-4 6.89 ± 0% 6.73 ± 0% -2.32% (p=0.000 n=5+4) Write/ordered-flate-4 6.89 ± 0% 6.69 ± 0% -2.84% (p=0.008 n=5+5) Write/ordered-zstd-4 6.17 ± 0% 6.16 ± 0% -0.02% (p=0.008 n=5+5) Write/unordered-gzip-4 6.89 ± 0% 6.73 ± 0% -2.32% (p=0.008 n=5+5) Write/unordered-flate-4 6.89 ± 0% 6.70 ± 0% -2.81% (p=0.000 n=4+5) Write/unordered-zstd-4 6.17 ± 0% 6.16 ± 0% -0.02% (p=0.008 n=5+5) name old alloc/op new alloc/op delta Write/unordered-lz4-4 18.8MB ± 4% 17.8MB ± 1% -5.28% (p=0.008 n=5+5) Read/gzip_66_kB-4 713kB ± 0% 735kB ± 1% +2.99% (p=0.008 n=5+5) Read/flate_66_kB-4 715kB ± 0% 733kB ± 0% +2.57% (p=0.008 n=5+5) Read/zstd_66_kB-4 423MB ± 0% 1MB ± 0% -99.65% (p=0.016 n=5+4) Read/gzip_262_kB-4 198kB ± 0% 204kB ± 0% +3.07% (p=0.016 n=4+5) Read/lz4-64k_262_kB-4 169kB ± 0% 169kB ± 0% -0.07% (p=0.000 n=5+4) Read/flate_262_kB-4 197kB ± 0% 205kB ± 0% +3.77% (p=0.008 n=5+5) Read/zstd_262_kB-4 553MB ± 0% 0MB ± 0% -99.92% (p=0.016 n=5+4) Read/gzip_524_kB-4 96.2kB ± 0% 98.8kB ± 0% +2.71% (p=0.016 n=4+5) Read/flate_524_kB-4 95.4kB ± 0% 98.7kB ± 0% +3.46% (p=0.008 n=5+5) Read/zstd_524_kB-4 786MB ± 0% 1MB ± 6% -99.93% (p=0.008 n=5+5) Read/sample_gzip_66_kB-4 686kB ± 0% 703kB ± 0% +2.48% (p=0.029 n=4+4) Read/sample_flate_66_kB-4 688kB ± 0% 710kB ± 1% +3.08% (p=0.016 n=4+5) Read/sample_zstd_66_kB-4 423MB ± 0% 2MB ± 0% -99.59% (p=0.016 n=5+4) Read/sample_gzip_262_kB-4 194kB ± 0% 200kB ± 0% +3.10% (p=0.016 n=4+5) Read/sample_flate_262_kB-4 194kB ± 0% 201kB ± 0% +3.77% (p=0.016 n=4+5) Read/sample_zstd_262_kB-4 553MB ± 0% 0MB ± 0% -99.91% (p=0.016 n=5+4) Read/sample_gzip_524_kB-4 95.4kB ± 0% 97.9kB ± 0% +2.65% (p=0.008 n=5+5) Read/sample_flate_524_kB-4 95.2kB ± 0% 98.8kB ± 0% +3.71% (p=0.016 n=5+4) Read/sample_zstd_524_kB-4 786MB ± 0% 1MB ± 0% -99.90% (p=0.008 n=5+5) name old allocs/op new allocs/op delta Write/ordered-gzip-4 701 ± 0% 718 ± 1% +2.40% (p=0.008 n=5+5) Write/ordered-lz4-256k-4 656 ± 0% 657 ± 0% +0.21% (p=0.048 n=5+5) Write/ordered-flate-4 626 ± 0% 634 ± 1% +1.37% (p=0.008 n=5+5) Write/ordered-zstd-4 2.07k ± 0% 1.44k ± 0% -30.49% (p=0.008 n=5+5) Write/unordered-gzip-4 418k ± 0% 428k ± 0% +2.39% (p=0.008 n=5+5) Write/unordered-flate-4 418k ± 0% 430k ± 0% +2.89% (p=0.008 n=5+5) Write/unordered-zstd-4 469k ± 0% 468k ± 0% -0.12% (p=0.008 n=5+5) Read/gzip_66_kB-4 13.2k ± 0% 13.5k ± 0% +2.54% (p=0.016 n=5+4) Read/lz4-256k_66_kB-4 13.7k ± 0% 13.7k ± 0% -0.01% (p=0.029 n=4+4) Read/flate_66_kB-4 13.2k ± 0% 13.6k ± 0% +2.72% (p=0.008 n=5+5) Read/zstd_66_kB-4 18.0k ± 1% 10.8k ± 0% -40.19% (p=0.008 n=5+5) Read/gzip_262_kB-4 4.61k ± 0% 4.76k ± 0% +3.17% (p=0.016 n=4+5) Read/flate_262_kB-4 4.61k ± 0% 4.78k ± 0% +3.62% (p=0.008 n=5+5) Read/zstd_262_kB-4 7.10k ± 2% 2.54k ± 0% -64.22% (p=0.016 n=5+4) Read/gzip_524_kB-4 2.62k ± 0% 2.69k ± 0% +2.36% (p=0.016 n=4+5) Read/flate_524_kB-4 2.62k ± 0% 2.70k ± 0% +3.09% (p=0.008 n=5+5) Read/zstd_524_kB-4 6.13k ± 2% 1.63k ± 0% -73.40% (p=0.008 n=5+5) Read/sample_gzip_66_kB-4 13.2k ± 0% 13.5k ± 0% +2.54% (p=0.029 n=4+4) Read/sample_flate_66_kB-4 13.2k ± 0% 13.6k ± 0% +2.88% (p=0.016 n=4+5) Read/sample_zstd_66_kB-4 18.0k ± 1% 10.8k ± 0% -40.00% (p=0.008 n=5+5) Read/sample_gzip_262_kB-4 4.63k ± 0% 4.77k ± 0% +3.16% (p=0.016 n=4+5) Read/sample_flate_262_kB-4 4.62k ± 0% 4.79k ± 0% +3.61% (p=0.016 n=4+5) Read/sample_zstd_262_kB-4 7.12k ± 1% 2.54k ± 0% -64.28% (p=0.008 n=5+5) Read/sample_gzip_524_kB-4 2.63k ± 0% 2.69k ± 0% +2.36% (p=0.008 n=5+5) Read/sample_flate_524_kB-4 2.63k ± 0% 2.72k ± 0% +3.31% (p=0.016 n=5+4) Read/sample_zstd_524_kB-4 6.11k ± 2% 1.64k ± 0% -73.23% (p=0.008 n=5+5) ```
lxwzy
pushed a commit
to lxwzy/loki
that referenced
this pull request
Nov 7, 2022
…me corrupt inputs (grafana#7264) **What this PR does / why we need it**: This is much faster, since we don't pay for byte-at-a-time reads from the buffer. For best performance with gzip and lz4, we still want a bufio.Reader. We don't need the pool of buffered readers now. Other compression formats have their own internal buffers so don't need extra buffering. **Checklist** - [x] Reviewed the `CONTRIBUTING.md` guide - NA Documentation added - NA Tests updated - [x] `CHANGELOG.md` updated - NA Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` (using changes from grafana#7246, ignoring lines that didn't change much) ``` name old time/op new time/op delta Write/ordered-lz4-256k-4 24.1ms ± 7% 22.1ms ± 2% -8.52% (p=0.008 n=5+5) Write/unordered-lz4-4 54.6ms ±29% 49.1ms ± 1% -10.04% (p=0.016 n=5+4) Read/none_66_kB-4 4.16ms ± 6% 2.54ms ± 2% -38.87% (p=0.008 n=5+5) Read/gzip_66_kB-4 177ms ± 4% 161ms ± 4% -9.09% (p=0.008 n=5+5) Read/lz4-64k_66_kB-4 59.5ms ± 2% 49.5ms ± 3% -16.80% (p=0.008 n=5+5) Read/lz4-256k_66_kB-4 62.9ms ± 0% 51.2ms ± 1% -18.58% (p=0.016 n=4+5) Read/lz4-1M_66_kB-4 62.9ms ± 2% 51.5ms ± 1% -18.03% (p=0.008 n=5+5) Read/lz4_66_kB-4 63.7ms ± 4% 51.2ms ± 1% -19.68% (p=0.008 n=5+5) Read/snappy_66_kB-4 51.1ms ± 2% 41.9ms ± 1% -17.95% (p=0.016 n=5+4) Read/flate_66_kB-4 168ms ± 4% 150ms ± 2% -10.49% (p=0.008 n=5+5) Read/zstd_66_kB-4 334ms ±16% 272ms ± 6% -18.52% (p=0.008 n=5+5) Read/none_262_kB-4 3.93ms ± 1% 2.48ms ± 2% -36.91% (p=0.008 n=5+5) Read/gzip_262_kB-4 159ms ± 5% 143ms ± 2% -10.12% (p=0.008 n=5+5) Read/lz4-64k_262_kB-4 53.5ms ± 1% 43.2ms ± 3% -19.35% (p=0.008 n=5+5) Read/lz4-256k_262_kB-4 55.8ms ± 1% 45.3ms ± 2% -18.75% (p=0.008 n=5+5) Read/lz4-1M_262_kB-4 57.5ms ± 2% 45.5ms ± 2% -20.91% (p=0.008 n=5+5) Read/lz4_262_kB-4 56.8ms ± 2% 45.3ms ± 2% -20.26% (p=0.008 n=5+5) Read/snappy_262_kB-4 46.0ms ± 2% 37.5ms ± 1% -18.57% (p=0.008 n=5+5) Read/flate_262_kB-4 153ms ± 3% 134ms ± 2% -12.33% (p=0.008 n=5+5) Read/none_524_kB-4 3.85ms ± 2% 2.46ms ± 2% -36.04% (p=0.008 n=5+5) Read/gzip_524_kB-4 128ms ± 2% 119ms ± 2% -7.64% (p=0.008 n=5+5) Read/lz4-64k_524_kB-4 44.6ms ± 3% 34.7ms ± 1% -22.19% (p=0.008 n=5+5) Read/lz4-256k_524_kB-4 45.4ms ± 1% 36.7ms ± 5% -19.07% (p=0.008 n=5+5) Read/lz4-1M_524_kB-4 47.3ms ± 1% 38.6ms ± 2% -18.36% (p=0.008 n=5+5) Read/lz4_524_kB-4 47.4ms ± 1% 38.0ms ± 1% -19.70% (p=0.008 n=5+5) Read/snappy_524_kB-4 37.3ms ± 1% 31.1ms ± 3% -16.47% (p=0.008 n=5+5) Read/flate_524_kB-4 122ms ± 1% 111ms ± 3% -9.04% (p=0.008 n=5+5) Read/sample_none_66_kB-4 7.81ms ± 1% 6.62ms ± 6% -15.24% (p=0.008 n=5+5) Read/sample_gzip_66_kB-4 237ms ± 2% 219ms ± 1% -7.57% (p=0.008 n=5+5) Read/sample_lz4-64k_66_kB-4 105ms ± 2% 92ms ± 1% -12.03% (p=0.008 n=5+5) Read/sample_lz4-256k_66_kB-4 109ms ± 1% 98ms ± 2% -9.65% (p=0.008 n=5+5) Read/sample_lz4-1M_66_kB-4 113ms ± 8% 98ms ± 2% -13.66% (p=0.008 n=5+5) Read/sample_lz4_66_kB-4 109ms ± 1% 99ms ± 2% -9.29% (p=0.008 n=5+5) Read/sample_snappy_66_kB-4 86.6ms ± 1% 77.2ms ± 6% -10.92% (p=0.008 n=5+5) Read/sample_flate_66_kB-4 236ms ± 8% 214ms ± 3% -9.11% (p=0.008 n=5+5) Read/sample_none_262_kB-4 7.70ms ± 2% 6.34ms ± 2% -17.67% (p=0.008 n=5+5) Read/sample_lz4-64k_262_kB-4 94.7ms ± 1% 84.4ms ± 2% -10.90% (p=0.008 n=5+5) Read/sample_lz4-256k_262_kB-4 98.4ms ± 1% 87.7ms ± 2% -10.93% (p=0.008 n=5+5) Read/sample_lz4-1M_262_kB-4 101ms ± 2% 90ms ± 2% -11.19% (p=0.008 n=5+5) Read/sample_lz4_262_kB-4 100ms ± 2% 90ms ± 3% -10.78% (p=0.008 n=5+5) Read/sample_snappy_262_kB-4 77.9ms ± 2% 68.7ms ± 2% -11.84% (p=0.008 n=5+5) Read/sample_flate_262_kB-4 211ms ± 2% 190ms ± 2% -9.86% (p=0.008 n=5+5) Read/sample_none_524_kB-4 7.82ms ± 2% 6.43ms ± 2% -17.80% (p=0.008 n=5+5) Read/sample_gzip_524_kB-4 176ms ± 1% 164ms ± 2% -6.80% (p=0.008 n=5+5) Read/sample_lz4-64k_524_kB-4 77.7ms ± 1% 69.8ms ± 4% -10.10% (p=0.008 n=5+5) Read/sample_lz4-256k_524_kB-4 81.7ms ± 2% 72.1ms ± 3% -11.79% (p=0.008 n=5+5) Read/sample_lz4-1M_524_kB-4 83.8ms ± 1% 74.4ms ± 1% -11.21% (p=0.008 n=5+5) Read/sample_lz4_524_kB-4 83.2ms ± 0% 76.1ms ± 8% -8.53% (p=0.016 n=4+5) Read/sample_snappy_524_kB-4 63.6ms ± 2% 56.5ms ± 2% -11.23% (p=0.008 n=5+5) Read/sample_flate_524_kB-4 172ms ± 2% 157ms ± 1% -8.88% (p=0.008 n=5+5) name old speed new speed delta Write/ordered-lz4-256k-4 687MB/s ± 6% 749MB/s ± 2% +9.09% (p=0.008 n=5+5) Write/unordered-lz4-4 313MB/s ±24% 341MB/s ± 1% +9.23% (p=0.016 n=5+4) Read/none_66_kB-4 1.82GB/s ± 6% 2.97GB/s ± 2% +63.42% (p=0.008 n=5+5) Read/gzip_66_kB-4 669MB/s ± 4% 736MB/s ± 4% +10.00% (p=0.008 n=5+5) Read/lz4-64k_66_kB-4 1.42GB/s ± 2% 1.71GB/s ± 3% +20.21% (p=0.008 n=5+5) Read/lz4-256k_66_kB-4 1.43GB/s ± 0% 1.76GB/s ± 1% +22.83% (p=0.016 n=4+5) Read/lz4-1M_66_kB-4 1.43GB/s ± 2% 1.74GB/s ± 1% +21.99% (p=0.008 n=5+5) Read/lz4_66_kB-4 1.41GB/s ± 4% 1.76GB/s ± 1% +24.44% (p=0.008 n=5+5) Read/snappy_66_kB-4 1.29GB/s ± 2% 1.57GB/s ± 1% +21.86% (p=0.016 n=5+4) Read/flate_66_kB-4 710MB/s ± 4% 793MB/s ± 2% +11.70% (p=0.008 n=5+5) Read/zstd_66_kB-4 425MB/s ±14% 517MB/s ± 6% +21.74% (p=0.008 n=5+5) Read/none_262_kB-4 1.93GB/s ± 1% 3.06GB/s ± 2% +58.54% (p=0.008 n=5+5) Read/gzip_262_kB-4 701MB/s ± 5% 779MB/s ± 2% +11.20% (p=0.008 n=5+5) Read/lz4-64k_262_kB-4 1.47GB/s ± 1% 1.82GB/s ± 3% +24.03% (p=0.008 n=5+5) Read/lz4-256k_262_kB-4 1.48GB/s ± 1% 1.82GB/s ± 2% +23.09% (p=0.008 n=5+5) Read/lz4-1M_262_kB-4 1.46GB/s ± 2% 1.84GB/s ± 2% +26.44% (p=0.008 n=5+5) Read/lz4_262_kB-4 1.48GB/s ± 2% 1.85GB/s ± 2% +25.42% (p=0.008 n=5+5) Read/snappy_262_kB-4 1.31GB/s ± 2% 1.61GB/s ± 1% +22.80% (p=0.008 n=5+5) Read/flate_262_kB-4 729MB/s ± 3% 831MB/s ± 2% +14.05% (p=0.008 n=5+5) Read/none_524_kB-4 1.97GB/s ± 2% 3.08GB/s ± 2% +56.33% (p=0.008 n=5+5) Read/gzip_524_kB-4 714MB/s ± 2% 773MB/s ± 2% +8.27% (p=0.008 n=5+5) Read/lz4-64k_524_kB-4 1.47GB/s ± 3% 1.88GB/s ± 1% +28.48% (p=0.008 n=5+5) Read/lz4-256k_524_kB-4 1.50GB/s ± 1% 1.86GB/s ± 5% +23.63% (p=0.008 n=5+5) Read/lz4-1M_524_kB-4 1.49GB/s ± 1% 1.83GB/s ± 2% +22.51% (p=0.008 n=5+5) Read/lz4_524_kB-4 1.49GB/s ± 1% 1.86GB/s ± 1% +24.53% (p=0.008 n=5+5) Read/snappy_524_kB-4 1.34GB/s ± 1% 1.60GB/s ± 3% +19.75% (p=0.008 n=5+5) Read/flate_524_kB-4 750MB/s ± 1% 824MB/s ± 3% +9.97% (p=0.008 n=5+5) Read/sample_none_66_kB-4 967MB/s ± 1% 1142MB/s ± 6% +18.13% (p=0.008 n=5+5) Read/sample_gzip_66_kB-4 501MB/s ± 2% 542MB/s ± 1% +8.19% (p=0.008 n=5+5) Read/sample_lz4-64k_66_kB-4 811MB/s ± 2% 922MB/s ± 1% +13.66% (p=0.008 n=5+5) Read/sample_lz4-256k_66_kB-4 825MB/s ± 1% 913MB/s ± 2% +10.68% (p=0.008 n=5+5) Read/sample_lz4-1M_66_kB-4 795MB/s ± 7% 920MB/s ± 1% +15.67% (p=0.008 n=5+5) Read/sample_lz4_66_kB-4 824MB/s ± 1% 909MB/s ± 2% +10.25% (p=0.008 n=5+5) Read/sample_snappy_66_kB-4 759MB/s ± 1% 852MB/s ± 6% +12.37% (p=0.008 n=5+5) Read/sample_flate_66_kB-4 506MB/s ± 7% 556MB/s ± 4% +9.88% (p=0.008 n=5+5) Read/sample_none_262_kB-4 983MB/s ± 2% 1194MB/s ± 2% +21.47% (p=0.008 n=5+5) Read/sample_lz4-64k_262_kB-4 830MB/s ± 1% 932MB/s ± 2% +12.24% (p=0.008 n=5+5) Read/sample_lz4-256k_262_kB-4 839MB/s ± 1% 942MB/s ± 2% +12.28% (p=0.008 n=5+5) Read/sample_lz4-1M_262_kB-4 829MB/s ± 2% 934MB/s ± 2% +12.60% (p=0.008 n=5+5) Read/sample_lz4_262_kB-4 836MB/s ± 2% 937MB/s ± 3% +12.11% (p=0.008 n=5+5) Read/sample_snappy_262_kB-4 774MB/s ± 2% 878MB/s ± 2% +13.44% (p=0.008 n=5+5) Read/sample_flate_262_kB-4 529MB/s ± 2% 587MB/s ± 2% +10.94% (p=0.008 n=5+5) Read/sample_none_524_kB-4 970MB/s ± 2% 1180MB/s ± 2% +21.64% (p=0.008 n=5+5) Read/sample_gzip_524_kB-4 521MB/s ± 1% 559MB/s ± 2% +7.31% (p=0.008 n=5+5) Read/sample_lz4-64k_524_kB-4 842MB/s ± 1% 937MB/s ± 4% +11.30% (p=0.008 n=5+5) Read/sample_lz4-256k_524_kB-4 834MB/s ± 2% 945MB/s ± 3% +13.37% (p=0.008 n=5+5) Read/sample_lz4-1M_524_kB-4 842MB/s ± 1% 949MB/s ± 1% +12.63% (p=0.008 n=5+5) Read/sample_lz4_524_kB-4 849MB/s ± 0% 929MB/s ± 7% +9.53% (p=0.016 n=4+5) Read/sample_snappy_524_kB-4 783MB/s ± 2% 882MB/s ± 2% +12.65% (p=0.008 n=5+5) Read/sample_flate_524_kB-4 534MB/s ± 2% 585MB/s ± 1% +9.73% (p=0.008 n=5+5) name old alloc/op new alloc/op delta Write/unordered-lz4-4 18.8MB ± 4% 17.2MB ± 7% -8.28% (p=0.008 n=5+5) Read/none_66_kB-4 41.9kB ± 0% 43.7kB ± 0% +4.41% (p=0.008 n=5+5) Read/gzip_66_kB-4 713kB ± 0% 741kB ± 0% +3.85% (p=0.016 n=5+4) Read/lz4-64k_66_kB-4 730kB ± 1% 748kB ± 0% +2.53% (p=0.016 n=5+4) Read/lz4-256k_66_kB-4 770kB ± 0% 806kB ± 3% +4.70% (p=0.016 n=4+5) Read/lz4-1M_66_kB-4 770kB ± 0% 812kB ± 4% +5.43% (p=0.016 n=4+5) Read/lz4_66_kB-4 770kB ± 0% 792kB ± 0% +2.83% (p=0.029 n=4+4) Read/snappy_66_kB-4 361kB ± 0% 376kB ± 0% +4.02% (p=0.008 n=5+5) Read/flate_66_kB-4 715kB ± 0% 743kB ± 0% +3.88% (p=0.016 n=5+4) Read/none_262_kB-4 11.2kB ± 0% 11.7kB ± 0% +4.12% (p=0.016 n=5+4) Read/gzip_262_kB-4 198kB ± 0% 203kB ± 0% +2.93% (p=0.016 n=4+5) Read/lz4-64k_262_kB-4 169kB ± 0% 174kB ± 0% +2.75% (p=0.008 n=5+5) Read/lz4-256k_262_kB-4 177kB ± 0% 182kB ± 0% +2.79% (p=0.008 n=5+5) Read/lz4-1M_262_kB-4 181kB ± 0% 186kB ± 0% +2.77% (p=0.008 n=5+5) Read/lz4_262_kB-4 181kB ± 0% 186kB ± 0% +2.73% (p=0.016 n=5+4) Read/snappy_262_kB-4 87.8kB ± 0% 90.1kB ± 0% +2.58% (p=0.008 n=5+5) Read/flate_262_kB-4 197kB ± 0% 203kB ± 0% +2.63% (p=0.016 n=5+4) Read/zstd_262_kB-4 553MB ± 0% 552MB ± 0% -0.15% (p=0.032 n=5+5) Read/none_524_kB-4 5.94kB ± 0% 6.16kB ± 0% +3.76% (p=0.016 n=5+4) Read/gzip_524_kB-4 96.2kB ± 0% 98.3kB ± 0% +2.16% (p=0.016 n=4+5) Read/lz4-64k_524_kB-4 70.9kB ± 0% 72.8kB ± 0% +2.71% (p=0.016 n=4+5) Read/lz4-256k_524_kB-4 73.8kB ± 0% 75.8kB ± 0% +2.68% (p=0.008 n=5+5) Read/lz4-1M_524_kB-4 76.5kB ± 0% 78.7kB ± 0% +2.77% (p=0.016 n=4+5) Read/lz4_524_kB-4 76.5kB ± 0% 78.6kB ± 0% +2.74% (p=0.016 n=5+4) Read/snappy_524_kB-4 39.1kB ± 0% 39.5kB ± 0% +1.19% (p=0.008 n=5+5) Read/flate_524_kB-4 95.4kB ± 0% 97.3kB ± 0% +2.06% (p=0.008 n=5+5) Read/sample_none_66_kB-4 39.9kB ± 0% 41.7kB ± 0% +4.62% (p=0.008 n=5+5) Read/sample_gzip_66_kB-4 686kB ± 0% 715kB ± 0% +4.23% (p=0.016 n=4+5) Read/sample_lz4-64k_66_kB-4 707kB ± 0% 728kB ± 0% +2.88% (p=0.016 n=5+4) Read/sample_lz4-256k_66_kB-4 749kB ± 0% 783kB ± 4% +4.65% (p=0.008 n=5+5) Read/sample_lz4_66_kB-4 748kB ± 0% 770kB ± 0% +2.91% (p=0.016 n=5+4) Read/sample_snappy_66_kB-4 350kB ± 0% 364kB ± 0% +4.04% (p=0.029 n=4+4) Read/sample_flate_66_kB-4 688kB ± 0% 716kB ± 0% +4.08% (p=0.029 n=4+4) Read/sample_none_262_kB-4 10.6kB ± 0% 11.0kB ± 0% +4.23% (p=0.016 n=5+4) Read/sample_gzip_262_kB-4 194kB ± 0% 199kB ± 0% +2.48% (p=0.016 n=4+5) Read/sample_lz4-64k_262_kB-4 165kB ± 0% 169kB ± 0% +2.86% (p=0.016 n=4+5) Read/sample_lz4-256k_262_kB-4 172kB ± 0% 177kB ± 0% +2.92% (p=0.016 n=4+5) Read/sample_lz4-1M_262_kB-4 176kB ± 0% 181kB ± 0% +2.91% (p=0.029 n=4+4) Read/sample_lz4_262_kB-4 176kB ± 0% 181kB ± 0% +2.81% (p=0.016 n=5+4) Read/sample_snappy_262_kB-4 88.1kB ± 0% 90.8kB ± 0% +3.11% (p=0.029 n=4+4) Read/sample_flate_262_kB-4 194kB ± 0% 198kB ± 0% +2.13% (p=0.029 n=4+4) Read/sample_none_524_kB-4 5.56kB ± 0% 5.76kB ± 0% +3.65% (p=0.016 n=5+4) Read/sample_gzip_524_kB-4 95.4kB ± 0% 96.9kB ± 0% +1.56% (p=0.008 n=5+5) Read/sample_lz4-64k_524_kB-4 69.0kB ± 0% 71.0kB ± 0% +2.87% (p=0.008 n=5+5) Read/sample_lz4-256k_524_kB-4 71.8kB ± 0% 73.8kB ± 0% +2.79% (p=0.016 n=4+5) Read/sample_lz4-1M_524_kB-4 74.5kB ± 0% 76.6kB ± 0% +2.75% (p=0.008 n=5+5) Read/sample_lz4_524_kB-4 74.5kB ± 0% 188.4kB ±89% +152.98% (p=0.008 n=5+5) Read/sample_snappy_524_kB-4 40.7kB ± 1% 41.3kB ± 1% +1.58% (p=0.008 n=5+5) Read/sample_flate_524_kB-4 95.2kB ± 0% 96.2kB ± 0% +0.95% (p=0.008 n=5+5) name old allocs/op new allocs/op delta Read/gzip_66_kB-4 13.2k ± 0% 13.2k ± 0% -0.04% (p=0.000 n=5+4) Read/zstd_66_kB-4 18.0k ± 1% 17.8k ± 1% -1.06% (p=0.032 n=5+5) Read/gzip_262_kB-4 4.61k ± 0% 4.61k ± 0% -0.11% (p=0.029 n=4+4) Read/flate_262_kB-4 4.61k ± 0% 4.60k ± 0% -0.11% (p=0.008 n=5+5) Read/zstd_262_kB-4 7.10k ± 2% 6.88k ± 3% -3.01% (p=0.008 n=5+5) Read/gzip_524_kB-4 2.62k ± 0% 2.62k ± 0% -0.06% (p=0.000 n=4+5) Read/zstd_524_kB-4 6.13k ± 2% 5.94k ± 2% -3.03% (p=0.024 n=5+5) Read/sample_flate_66_kB-4 13.2k ± 0% 13.2k ± 0% -0.01% (p=0.029 n=4+4) Read/sample_flate_262_kB-4 4.62k ± 0% 4.61k ± 0% -0.22% (p=0.029 n=4+4) Read/sample_zstd_262_kB-4 7.12k ± 1% 6.97k ± 1% -2.11% (p=0.008 n=5+5) Read/sample_gzip_524_kB-4 2.63k ± 0% 2.63k ± 0% -0.11% (p=0.008 n=5+5) Read/sample_flate_524_kB-4 2.63k ± 0% 2.62k ± 0% -0.15% (p=0.008 n=5+5) Read/sample_zstd_524_kB-4 6.11k ± 2% 5.89k ± 1% -3.68% (p=0.016 n=5+4) ``` Co-authored-by: Owen Diehl <[email protected]>
lxwzy
pushed a commit
to lxwzy/loki
that referenced
this pull request
Nov 7, 2022
**What this PR does / why we need it**: ### BenchmarkWrite Use Go runtime facilities to report MB/s for each operation and compression ratio. Before: ``` BenchmarkWrite/ordered-none-4 1018 1104885 ns/op 1431962 B/op 33 allocs/op BenchmarkWrite/ordered-gzip-4 12 87333201 ns/op 3516699 B/op 696 allocs/op BenchmarkWrite/ordered-lz4-64k-4 51 21483048 ns/op 3095117 B/op 649 allocs/op ``` After: ``` BenchmarkWrite/ordered-none-4 841 1202007 ns/op 1054353.53 MB/s 101.9 %compressed 1569953 B/op 34 allocs/op BenchmarkWrite/ordered-gzip-4 13 86299699 ns/op 3357.25 MB/s 6.891 %compressed 3778896 B/op 702 allocs/op BenchmarkWrite/ordered-lz4-64k-4 49 22332214 ns/op 34107.61 MB/s 9.880 %compressed 3407522 B/op 661 allocs/op ``` Stop collecting all compressed chunks, as this blows up the memory of the benchmark and creates unrealistic test conditions for the later encoding. Before: `4724260 maxresident KB`; after: `375804 maxresident KB` ### BenchmarkRead Replace reporting of MB/s which gave several values for each encoding as Go hunted for the right number of benchmark iterations. Add stats to the context for decoding, otherwise a new stats object is created each time round the loop. Re-order so all unsampled results come before all sampled results. Before: ``` BenchmarkRead/none_66_kB-4 278 4555157 ns/op 131744 B/op 717 allocs/op BenchmarkRead/sample_none_66_kB-4 142 8238861 ns/op 129771 B/op 717 allocs/op BenchmarkRead/gzip_66_kB-4 6 179445187 ns/op 2100498 B/op 15010 allocs/op BenchmarkRead/sample_gzip_66_kB-4 4 261040057 ns/op 2076030 B/op 15028 allocs/op BenchmarkRead/lz4-64k_66_kB-4 19 62855240 ns/op 1722979 B/op 14219 allocs/op BenchmarkRead/sample_lz4-64k_66_kB-4 10 115577796 ns/op 1706040 B/op 14220 allocs/op BenchmarkRead/lz4-256k_66_kB-4 18 66144317 ns/op 1880851 B/op 15078 allocs/op BenchmarkRead/sample_lz4-256k_66_kB-4 9 118405993 ns/op 1800396 B/op 15077 allocs/op BenchmarkRead/lz4-1M_66_kB-4 18 67832189 ns/op 1821806 B/op 15076 allocs/op ... none_524 kB: 1828.92 MB/s none_524 kB: 1809.33 MB/s none_262 kB: 1790.79 MB/s none_66 kB: 1673.84 MB/s ... gzip_66 kB: 647.35 MB/s ... ``` After: ``` BenchmarkRead/none_66_kB-4 286 4377585 ns/op 1725.23 MB/s 41896 B/op 600 allocs/op BenchmarkRead/gzip_66_kB-4 6 180770638 ns/op 655.82 MB/s 723426 B/op 13265 allocs/op BenchmarkRead/lz4-64k_66_kB-4 19 65380524 ns/op 1296.10 MB/s 727571 B/op 12925 allocs/op BenchmarkRead/lz4-256k_66_kB-4 16 63737292 ns/op 1409.75 MB/s 786994 B/op 13707 allocs/op BenchmarkRead/lz4-1M_66_kB-4 18 65963825 ns/op 1362.17 MB/s 770441 B/op 13707 allocs/op BenchmarkRead/lz4_66_kB-4 18 64955415 ns/op 1383.31 MB/s 1003277 B/op 13707 allocs/op BenchmarkRead/snappy_66_kB-4 20 51588959 ns/op 1273.38 MB/s 361872 B/op 5020 allocs/op BenchmarkRead/flate_66_kB-4 6 172147502 ns/op 691.74 MB/s 715706 B/op 13242 allocs/op BenchmarkRead/zstd_66_kB-4 3 359473273 ns/op 390.07 MB/s 427247248 B/op 18551 allocs/op ... BenchmarkRead/sample_none_66_kB-4 135 8533283 ns/op 885.04 MB/s 39866 B/op 600 allocs/op BenchmarkRead/sample_gzip_66_kB-4 5 243016949 ns/op 487.84 MB/s 686358 B/op 13212 allocs/op BenchmarkRead/sample_lz4-64k_66_kB-4 10 114330032 ns/op 741.18 MB/s 707104 B/op 12926 allocs/op BenchmarkRead/sample_lz4-256k_66_kB-4 9 117294928 ns/op 766.05 MB/s 778461 B/op 13709 allocs/op ... ``` Note "MB" in previous code was 2^^20 while Go uses 10^6. **Checklist** - [x] Reviewed the `CONTRIBUTING.md` guide - NA Documentation added - [x] Tests updated - NA `CHANGELOG.md` updated - not user-facing - NA Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`
changhyuni
pushed a commit
to changhyuni/loki
that referenced
this pull request
Nov 8, 2022
**What this PR does / why we need it**: Bring in various improvements from upstream, particularly klauspost/compress#668 **Checklist** - [x] Reviewed the `CONTRIBUTING.md` guide - NA Documentation added - NA Tests updated - [x] `CHANGELOG.md` updated - NA Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` **Benchmarks** (using changes from grafana#7246, ignoring lines that didn't change much) ``` name old time/op new time/op delta Write/ordered-flate-4 81.5ms ± 5% 78.7ms ± 2% -3.39% (p=0.016 n=5+5) Read/none_66_kB-4 4.16ms ± 6% 3.93ms ± 4% -5.51% (p=0.032 n=5+5) Read/flate_66_kB-4 168ms ± 4% 179ms ±11% +6.50% (p=0.032 n=5+5) Read/zstd_66_kB-4 334ms ±16% 152ms ± 3% -54.57% (p=0.008 n=5+5) Read/none_262_kB-4 3.93ms ± 1% 3.81ms ± 2% -2.91% (p=0.008 n=5+5) Read/zstd_262_kB-4 369ms ± 8% 131ms ± 1% -64.62% (p=0.008 n=5+5) Read/zstd_524_kB-4 466ms ±17% 146ms ± 2% -68.68% (p=0.008 n=5+5) Read/sample_zstd_66_kB-4 420ms ±18% 224ms ± 1% -46.69% (p=0.016 n=5+4) Read/sample_gzip_262_kB-4 214ms ± 1% 218ms ± 1% +1.90% (p=0.016 n=5+5) Read/sample_zstd_262_kB-4 418ms ±21% 200ms ± 2% -52.15% (p=0.016 n=5+4) Read/sample_zstd_524_kB-4 576ms ±35% 234ms ± 1% -59.42% (p=0.008 n=5+5) name old speed new speed delta Write/ordered-gzip-4 267MB/s ±10% 284MB/s ± 2% +6.51% (p=0.008 n=5+5) Write/ordered-flate-4 274MB/s ± 4% 291MB/s ± 2% +6.49% (p=0.008 n=5+5) Read/none_66_kB-4 1.82GB/s ± 6% 1.92GB/s ± 4% +5.78% (p=0.032 n=5+5) Read/gzip_66_kB-4 669MB/s ± 4% 698MB/s ± 1% +4.30% (p=0.008 n=5+5) Read/zstd_66_kB-4 425MB/s ±14% 930MB/s ± 3% +119.12% (p=0.008 n=5+5) Read/none_262_kB-4 1.93GB/s ± 1% 1.99GB/s ± 2% +3.00% (p=0.008 n=5+5) Read/gzip_262_kB-4 701MB/s ± 5% 721MB/s ± 2% +2.96% (p=0.032 n=5+5) Read/zstd_262_kB-4 357MB/s ± 8% 1009MB/s ± 1% +182.89% (p=0.008 n=5+5) Read/flate_524_kB-4 750MB/s ± 1% 763MB/s ± 1% +1.74% (p=0.008 n=5+5) Read/zstd_524_kB-4 365MB/s ±18% 1149MB/s ± 2% +215.34% (p=0.008 n=5+5) Read/sample_zstd_66_kB-4 341MB/s ±17% 616MB/s ± 9% +80.67% (p=0.008 n=5+5) Read/sample_zstd_262_kB-4 318MB/s ±18% 658MB/s ± 2% +107.08% (p=0.016 n=5+4) Read/sample_gzip_524_kB-4 521MB/s ± 1% 530MB/s ± 1% +1.72% (p=0.032 n=5+5) Read/sample_zstd_524_kB-4 299MB/s ±28% 718MB/s ± 1% +140.23% (p=0.008 n=5+5) name old %compressed new %compressed delta Write/ordered-gzip-4 6.89 ± 0% 6.73 ± 0% -2.32% (p=0.000 n=5+4) Write/ordered-flate-4 6.89 ± 0% 6.69 ± 0% -2.84% (p=0.008 n=5+5) Write/ordered-zstd-4 6.17 ± 0% 6.16 ± 0% -0.02% (p=0.008 n=5+5) Write/unordered-gzip-4 6.89 ± 0% 6.73 ± 0% -2.32% (p=0.008 n=5+5) Write/unordered-flate-4 6.89 ± 0% 6.70 ± 0% -2.81% (p=0.000 n=4+5) Write/unordered-zstd-4 6.17 ± 0% 6.16 ± 0% -0.02% (p=0.008 n=5+5) name old alloc/op new alloc/op delta Write/unordered-lz4-4 18.8MB ± 4% 17.8MB ± 1% -5.28% (p=0.008 n=5+5) Read/gzip_66_kB-4 713kB ± 0% 735kB ± 1% +2.99% (p=0.008 n=5+5) Read/flate_66_kB-4 715kB ± 0% 733kB ± 0% +2.57% (p=0.008 n=5+5) Read/zstd_66_kB-4 423MB ± 0% 1MB ± 0% -99.65% (p=0.016 n=5+4) Read/gzip_262_kB-4 198kB ± 0% 204kB ± 0% +3.07% (p=0.016 n=4+5) Read/lz4-64k_262_kB-4 169kB ± 0% 169kB ± 0% -0.07% (p=0.000 n=5+4) Read/flate_262_kB-4 197kB ± 0% 205kB ± 0% +3.77% (p=0.008 n=5+5) Read/zstd_262_kB-4 553MB ± 0% 0MB ± 0% -99.92% (p=0.016 n=5+4) Read/gzip_524_kB-4 96.2kB ± 0% 98.8kB ± 0% +2.71% (p=0.016 n=4+5) Read/flate_524_kB-4 95.4kB ± 0% 98.7kB ± 0% +3.46% (p=0.008 n=5+5) Read/zstd_524_kB-4 786MB ± 0% 1MB ± 6% -99.93% (p=0.008 n=5+5) Read/sample_gzip_66_kB-4 686kB ± 0% 703kB ± 0% +2.48% (p=0.029 n=4+4) Read/sample_flate_66_kB-4 688kB ± 0% 710kB ± 1% +3.08% (p=0.016 n=4+5) Read/sample_zstd_66_kB-4 423MB ± 0% 2MB ± 0% -99.59% (p=0.016 n=5+4) Read/sample_gzip_262_kB-4 194kB ± 0% 200kB ± 0% +3.10% (p=0.016 n=4+5) Read/sample_flate_262_kB-4 194kB ± 0% 201kB ± 0% +3.77% (p=0.016 n=4+5) Read/sample_zstd_262_kB-4 553MB ± 0% 0MB ± 0% -99.91% (p=0.016 n=5+4) Read/sample_gzip_524_kB-4 95.4kB ± 0% 97.9kB ± 0% +2.65% (p=0.008 n=5+5) Read/sample_flate_524_kB-4 95.2kB ± 0% 98.8kB ± 0% +3.71% (p=0.016 n=5+4) Read/sample_zstd_524_kB-4 786MB ± 0% 1MB ± 0% -99.90% (p=0.008 n=5+5) name old allocs/op new allocs/op delta Write/ordered-gzip-4 701 ± 0% 718 ± 1% +2.40% (p=0.008 n=5+5) Write/ordered-lz4-256k-4 656 ± 0% 657 ± 0% +0.21% (p=0.048 n=5+5) Write/ordered-flate-4 626 ± 0% 634 ± 1% +1.37% (p=0.008 n=5+5) Write/ordered-zstd-4 2.07k ± 0% 1.44k ± 0% -30.49% (p=0.008 n=5+5) Write/unordered-gzip-4 418k ± 0% 428k ± 0% +2.39% (p=0.008 n=5+5) Write/unordered-flate-4 418k ± 0% 430k ± 0% +2.89% (p=0.008 n=5+5) Write/unordered-zstd-4 469k ± 0% 468k ± 0% -0.12% (p=0.008 n=5+5) Read/gzip_66_kB-4 13.2k ± 0% 13.5k ± 0% +2.54% (p=0.016 n=5+4) Read/lz4-256k_66_kB-4 13.7k ± 0% 13.7k ± 0% -0.01% (p=0.029 n=4+4) Read/flate_66_kB-4 13.2k ± 0% 13.6k ± 0% +2.72% (p=0.008 n=5+5) Read/zstd_66_kB-4 18.0k ± 1% 10.8k ± 0% -40.19% (p=0.008 n=5+5) Read/gzip_262_kB-4 4.61k ± 0% 4.76k ± 0% +3.17% (p=0.016 n=4+5) Read/flate_262_kB-4 4.61k ± 0% 4.78k ± 0% +3.62% (p=0.008 n=5+5) Read/zstd_262_kB-4 7.10k ± 2% 2.54k ± 0% -64.22% (p=0.016 n=5+4) Read/gzip_524_kB-4 2.62k ± 0% 2.69k ± 0% +2.36% (p=0.016 n=4+5) Read/flate_524_kB-4 2.62k ± 0% 2.70k ± 0% +3.09% (p=0.008 n=5+5) Read/zstd_524_kB-4 6.13k ± 2% 1.63k ± 0% -73.40% (p=0.008 n=5+5) Read/sample_gzip_66_kB-4 13.2k ± 0% 13.5k ± 0% +2.54% (p=0.029 n=4+4) Read/sample_flate_66_kB-4 13.2k ± 0% 13.6k ± 0% +2.88% (p=0.016 n=4+5) Read/sample_zstd_66_kB-4 18.0k ± 1% 10.8k ± 0% -40.00% (p=0.008 n=5+5) Read/sample_gzip_262_kB-4 4.63k ± 0% 4.77k ± 0% +3.16% (p=0.016 n=4+5) Read/sample_flate_262_kB-4 4.62k ± 0% 4.79k ± 0% +3.61% (p=0.016 n=4+5) Read/sample_zstd_262_kB-4 7.12k ± 1% 2.54k ± 0% -64.28% (p=0.008 n=5+5) Read/sample_gzip_524_kB-4 2.63k ± 0% 2.69k ± 0% +2.36% (p=0.008 n=5+5) Read/sample_flate_524_kB-4 2.63k ± 0% 2.72k ± 0% +3.31% (p=0.016 n=5+4) Read/sample_zstd_524_kB-4 6.11k ± 2% 1.64k ± 0% -73.23% (p=0.008 n=5+5) ```
changhyuni
pushed a commit
to changhyuni/loki
that referenced
this pull request
Nov 8, 2022
…me corrupt inputs (grafana#7264) **What this PR does / why we need it**: This is much faster, since we don't pay for byte-at-a-time reads from the buffer. For best performance with gzip and lz4, we still want a bufio.Reader. We don't need the pool of buffered readers now. Other compression formats have their own internal buffers so don't need extra buffering. **Checklist** - [x] Reviewed the `CONTRIBUTING.md` guide - NA Documentation added - NA Tests updated - [x] `CHANGELOG.md` updated - NA Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` (using changes from grafana#7246, ignoring lines that didn't change much) ``` name old time/op new time/op delta Write/ordered-lz4-256k-4 24.1ms ± 7% 22.1ms ± 2% -8.52% (p=0.008 n=5+5) Write/unordered-lz4-4 54.6ms ±29% 49.1ms ± 1% -10.04% (p=0.016 n=5+4) Read/none_66_kB-4 4.16ms ± 6% 2.54ms ± 2% -38.87% (p=0.008 n=5+5) Read/gzip_66_kB-4 177ms ± 4% 161ms ± 4% -9.09% (p=0.008 n=5+5) Read/lz4-64k_66_kB-4 59.5ms ± 2% 49.5ms ± 3% -16.80% (p=0.008 n=5+5) Read/lz4-256k_66_kB-4 62.9ms ± 0% 51.2ms ± 1% -18.58% (p=0.016 n=4+5) Read/lz4-1M_66_kB-4 62.9ms ± 2% 51.5ms ± 1% -18.03% (p=0.008 n=5+5) Read/lz4_66_kB-4 63.7ms ± 4% 51.2ms ± 1% -19.68% (p=0.008 n=5+5) Read/snappy_66_kB-4 51.1ms ± 2% 41.9ms ± 1% -17.95% (p=0.016 n=5+4) Read/flate_66_kB-4 168ms ± 4% 150ms ± 2% -10.49% (p=0.008 n=5+5) Read/zstd_66_kB-4 334ms ±16% 272ms ± 6% -18.52% (p=0.008 n=5+5) Read/none_262_kB-4 3.93ms ± 1% 2.48ms ± 2% -36.91% (p=0.008 n=5+5) Read/gzip_262_kB-4 159ms ± 5% 143ms ± 2% -10.12% (p=0.008 n=5+5) Read/lz4-64k_262_kB-4 53.5ms ± 1% 43.2ms ± 3% -19.35% (p=0.008 n=5+5) Read/lz4-256k_262_kB-4 55.8ms ± 1% 45.3ms ± 2% -18.75% (p=0.008 n=5+5) Read/lz4-1M_262_kB-4 57.5ms ± 2% 45.5ms ± 2% -20.91% (p=0.008 n=5+5) Read/lz4_262_kB-4 56.8ms ± 2% 45.3ms ± 2% -20.26% (p=0.008 n=5+5) Read/snappy_262_kB-4 46.0ms ± 2% 37.5ms ± 1% -18.57% (p=0.008 n=5+5) Read/flate_262_kB-4 153ms ± 3% 134ms ± 2% -12.33% (p=0.008 n=5+5) Read/none_524_kB-4 3.85ms ± 2% 2.46ms ± 2% -36.04% (p=0.008 n=5+5) Read/gzip_524_kB-4 128ms ± 2% 119ms ± 2% -7.64% (p=0.008 n=5+5) Read/lz4-64k_524_kB-4 44.6ms ± 3% 34.7ms ± 1% -22.19% (p=0.008 n=5+5) Read/lz4-256k_524_kB-4 45.4ms ± 1% 36.7ms ± 5% -19.07% (p=0.008 n=5+5) Read/lz4-1M_524_kB-4 47.3ms ± 1% 38.6ms ± 2% -18.36% (p=0.008 n=5+5) Read/lz4_524_kB-4 47.4ms ± 1% 38.0ms ± 1% -19.70% (p=0.008 n=5+5) Read/snappy_524_kB-4 37.3ms ± 1% 31.1ms ± 3% -16.47% (p=0.008 n=5+5) Read/flate_524_kB-4 122ms ± 1% 111ms ± 3% -9.04% (p=0.008 n=5+5) Read/sample_none_66_kB-4 7.81ms ± 1% 6.62ms ± 6% -15.24% (p=0.008 n=5+5) Read/sample_gzip_66_kB-4 237ms ± 2% 219ms ± 1% -7.57% (p=0.008 n=5+5) Read/sample_lz4-64k_66_kB-4 105ms ± 2% 92ms ± 1% -12.03% (p=0.008 n=5+5) Read/sample_lz4-256k_66_kB-4 109ms ± 1% 98ms ± 2% -9.65% (p=0.008 n=5+5) Read/sample_lz4-1M_66_kB-4 113ms ± 8% 98ms ± 2% -13.66% (p=0.008 n=5+5) Read/sample_lz4_66_kB-4 109ms ± 1% 99ms ± 2% -9.29% (p=0.008 n=5+5) Read/sample_snappy_66_kB-4 86.6ms ± 1% 77.2ms ± 6% -10.92% (p=0.008 n=5+5) Read/sample_flate_66_kB-4 236ms ± 8% 214ms ± 3% -9.11% (p=0.008 n=5+5) Read/sample_none_262_kB-4 7.70ms ± 2% 6.34ms ± 2% -17.67% (p=0.008 n=5+5) Read/sample_lz4-64k_262_kB-4 94.7ms ± 1% 84.4ms ± 2% -10.90% (p=0.008 n=5+5) Read/sample_lz4-256k_262_kB-4 98.4ms ± 1% 87.7ms ± 2% -10.93% (p=0.008 n=5+5) Read/sample_lz4-1M_262_kB-4 101ms ± 2% 90ms ± 2% -11.19% (p=0.008 n=5+5) Read/sample_lz4_262_kB-4 100ms ± 2% 90ms ± 3% -10.78% (p=0.008 n=5+5) Read/sample_snappy_262_kB-4 77.9ms ± 2% 68.7ms ± 2% -11.84% (p=0.008 n=5+5) Read/sample_flate_262_kB-4 211ms ± 2% 190ms ± 2% -9.86% (p=0.008 n=5+5) Read/sample_none_524_kB-4 7.82ms ± 2% 6.43ms ± 2% -17.80% (p=0.008 n=5+5) Read/sample_gzip_524_kB-4 176ms ± 1% 164ms ± 2% -6.80% (p=0.008 n=5+5) Read/sample_lz4-64k_524_kB-4 77.7ms ± 1% 69.8ms ± 4% -10.10% (p=0.008 n=5+5) Read/sample_lz4-256k_524_kB-4 81.7ms ± 2% 72.1ms ± 3% -11.79% (p=0.008 n=5+5) Read/sample_lz4-1M_524_kB-4 83.8ms ± 1% 74.4ms ± 1% -11.21% (p=0.008 n=5+5) Read/sample_lz4_524_kB-4 83.2ms ± 0% 76.1ms ± 8% -8.53% (p=0.016 n=4+5) Read/sample_snappy_524_kB-4 63.6ms ± 2% 56.5ms ± 2% -11.23% (p=0.008 n=5+5) Read/sample_flate_524_kB-4 172ms ± 2% 157ms ± 1% -8.88% (p=0.008 n=5+5) name old speed new speed delta Write/ordered-lz4-256k-4 687MB/s ± 6% 749MB/s ± 2% +9.09% (p=0.008 n=5+5) Write/unordered-lz4-4 313MB/s ±24% 341MB/s ± 1% +9.23% (p=0.016 n=5+4) Read/none_66_kB-4 1.82GB/s ± 6% 2.97GB/s ± 2% +63.42% (p=0.008 n=5+5) Read/gzip_66_kB-4 669MB/s ± 4% 736MB/s ± 4% +10.00% (p=0.008 n=5+5) Read/lz4-64k_66_kB-4 1.42GB/s ± 2% 1.71GB/s ± 3% +20.21% (p=0.008 n=5+5) Read/lz4-256k_66_kB-4 1.43GB/s ± 0% 1.76GB/s ± 1% +22.83% (p=0.016 n=4+5) Read/lz4-1M_66_kB-4 1.43GB/s ± 2% 1.74GB/s ± 1% +21.99% (p=0.008 n=5+5) Read/lz4_66_kB-4 1.41GB/s ± 4% 1.76GB/s ± 1% +24.44% (p=0.008 n=5+5) Read/snappy_66_kB-4 1.29GB/s ± 2% 1.57GB/s ± 1% +21.86% (p=0.016 n=5+4) Read/flate_66_kB-4 710MB/s ± 4% 793MB/s ± 2% +11.70% (p=0.008 n=5+5) Read/zstd_66_kB-4 425MB/s ±14% 517MB/s ± 6% +21.74% (p=0.008 n=5+5) Read/none_262_kB-4 1.93GB/s ± 1% 3.06GB/s ± 2% +58.54% (p=0.008 n=5+5) Read/gzip_262_kB-4 701MB/s ± 5% 779MB/s ± 2% +11.20% (p=0.008 n=5+5) Read/lz4-64k_262_kB-4 1.47GB/s ± 1% 1.82GB/s ± 3% +24.03% (p=0.008 n=5+5) Read/lz4-256k_262_kB-4 1.48GB/s ± 1% 1.82GB/s ± 2% +23.09% (p=0.008 n=5+5) Read/lz4-1M_262_kB-4 1.46GB/s ± 2% 1.84GB/s ± 2% +26.44% (p=0.008 n=5+5) Read/lz4_262_kB-4 1.48GB/s ± 2% 1.85GB/s ± 2% +25.42% (p=0.008 n=5+5) Read/snappy_262_kB-4 1.31GB/s ± 2% 1.61GB/s ± 1% +22.80% (p=0.008 n=5+5) Read/flate_262_kB-4 729MB/s ± 3% 831MB/s ± 2% +14.05% (p=0.008 n=5+5) Read/none_524_kB-4 1.97GB/s ± 2% 3.08GB/s ± 2% +56.33% (p=0.008 n=5+5) Read/gzip_524_kB-4 714MB/s ± 2% 773MB/s ± 2% +8.27% (p=0.008 n=5+5) Read/lz4-64k_524_kB-4 1.47GB/s ± 3% 1.88GB/s ± 1% +28.48% (p=0.008 n=5+5) Read/lz4-256k_524_kB-4 1.50GB/s ± 1% 1.86GB/s ± 5% +23.63% (p=0.008 n=5+5) Read/lz4-1M_524_kB-4 1.49GB/s ± 1% 1.83GB/s ± 2% +22.51% (p=0.008 n=5+5) Read/lz4_524_kB-4 1.49GB/s ± 1% 1.86GB/s ± 1% +24.53% (p=0.008 n=5+5) Read/snappy_524_kB-4 1.34GB/s ± 1% 1.60GB/s ± 3% +19.75% (p=0.008 n=5+5) Read/flate_524_kB-4 750MB/s ± 1% 824MB/s ± 3% +9.97% (p=0.008 n=5+5) Read/sample_none_66_kB-4 967MB/s ± 1% 1142MB/s ± 6% +18.13% (p=0.008 n=5+5) Read/sample_gzip_66_kB-4 501MB/s ± 2% 542MB/s ± 1% +8.19% (p=0.008 n=5+5) Read/sample_lz4-64k_66_kB-4 811MB/s ± 2% 922MB/s ± 1% +13.66% (p=0.008 n=5+5) Read/sample_lz4-256k_66_kB-4 825MB/s ± 1% 913MB/s ± 2% +10.68% (p=0.008 n=5+5) Read/sample_lz4-1M_66_kB-4 795MB/s ± 7% 920MB/s ± 1% +15.67% (p=0.008 n=5+5) Read/sample_lz4_66_kB-4 824MB/s ± 1% 909MB/s ± 2% +10.25% (p=0.008 n=5+5) Read/sample_snappy_66_kB-4 759MB/s ± 1% 852MB/s ± 6% +12.37% (p=0.008 n=5+5) Read/sample_flate_66_kB-4 506MB/s ± 7% 556MB/s ± 4% +9.88% (p=0.008 n=5+5) Read/sample_none_262_kB-4 983MB/s ± 2% 1194MB/s ± 2% +21.47% (p=0.008 n=5+5) Read/sample_lz4-64k_262_kB-4 830MB/s ± 1% 932MB/s ± 2% +12.24% (p=0.008 n=5+5) Read/sample_lz4-256k_262_kB-4 839MB/s ± 1% 942MB/s ± 2% +12.28% (p=0.008 n=5+5) Read/sample_lz4-1M_262_kB-4 829MB/s ± 2% 934MB/s ± 2% +12.60% (p=0.008 n=5+5) Read/sample_lz4_262_kB-4 836MB/s ± 2% 937MB/s ± 3% +12.11% (p=0.008 n=5+5) Read/sample_snappy_262_kB-4 774MB/s ± 2% 878MB/s ± 2% +13.44% (p=0.008 n=5+5) Read/sample_flate_262_kB-4 529MB/s ± 2% 587MB/s ± 2% +10.94% (p=0.008 n=5+5) Read/sample_none_524_kB-4 970MB/s ± 2% 1180MB/s ± 2% +21.64% (p=0.008 n=5+5) Read/sample_gzip_524_kB-4 521MB/s ± 1% 559MB/s ± 2% +7.31% (p=0.008 n=5+5) Read/sample_lz4-64k_524_kB-4 842MB/s ± 1% 937MB/s ± 4% +11.30% (p=0.008 n=5+5) Read/sample_lz4-256k_524_kB-4 834MB/s ± 2% 945MB/s ± 3% +13.37% (p=0.008 n=5+5) Read/sample_lz4-1M_524_kB-4 842MB/s ± 1% 949MB/s ± 1% +12.63% (p=0.008 n=5+5) Read/sample_lz4_524_kB-4 849MB/s ± 0% 929MB/s ± 7% +9.53% (p=0.016 n=4+5) Read/sample_snappy_524_kB-4 783MB/s ± 2% 882MB/s ± 2% +12.65% (p=0.008 n=5+5) Read/sample_flate_524_kB-4 534MB/s ± 2% 585MB/s ± 1% +9.73% (p=0.008 n=5+5) name old alloc/op new alloc/op delta Write/unordered-lz4-4 18.8MB ± 4% 17.2MB ± 7% -8.28% (p=0.008 n=5+5) Read/none_66_kB-4 41.9kB ± 0% 43.7kB ± 0% +4.41% (p=0.008 n=5+5) Read/gzip_66_kB-4 713kB ± 0% 741kB ± 0% +3.85% (p=0.016 n=5+4) Read/lz4-64k_66_kB-4 730kB ± 1% 748kB ± 0% +2.53% (p=0.016 n=5+4) Read/lz4-256k_66_kB-4 770kB ± 0% 806kB ± 3% +4.70% (p=0.016 n=4+5) Read/lz4-1M_66_kB-4 770kB ± 0% 812kB ± 4% +5.43% (p=0.016 n=4+5) Read/lz4_66_kB-4 770kB ± 0% 792kB ± 0% +2.83% (p=0.029 n=4+4) Read/snappy_66_kB-4 361kB ± 0% 376kB ± 0% +4.02% (p=0.008 n=5+5) Read/flate_66_kB-4 715kB ± 0% 743kB ± 0% +3.88% (p=0.016 n=5+4) Read/none_262_kB-4 11.2kB ± 0% 11.7kB ± 0% +4.12% (p=0.016 n=5+4) Read/gzip_262_kB-4 198kB ± 0% 203kB ± 0% +2.93% (p=0.016 n=4+5) Read/lz4-64k_262_kB-4 169kB ± 0% 174kB ± 0% +2.75% (p=0.008 n=5+5) Read/lz4-256k_262_kB-4 177kB ± 0% 182kB ± 0% +2.79% (p=0.008 n=5+5) Read/lz4-1M_262_kB-4 181kB ± 0% 186kB ± 0% +2.77% (p=0.008 n=5+5) Read/lz4_262_kB-4 181kB ± 0% 186kB ± 0% +2.73% (p=0.016 n=5+4) Read/snappy_262_kB-4 87.8kB ± 0% 90.1kB ± 0% +2.58% (p=0.008 n=5+5) Read/flate_262_kB-4 197kB ± 0% 203kB ± 0% +2.63% (p=0.016 n=5+4) Read/zstd_262_kB-4 553MB ± 0% 552MB ± 0% -0.15% (p=0.032 n=5+5) Read/none_524_kB-4 5.94kB ± 0% 6.16kB ± 0% +3.76% (p=0.016 n=5+4) Read/gzip_524_kB-4 96.2kB ± 0% 98.3kB ± 0% +2.16% (p=0.016 n=4+5) Read/lz4-64k_524_kB-4 70.9kB ± 0% 72.8kB ± 0% +2.71% (p=0.016 n=4+5) Read/lz4-256k_524_kB-4 73.8kB ± 0% 75.8kB ± 0% +2.68% (p=0.008 n=5+5) Read/lz4-1M_524_kB-4 76.5kB ± 0% 78.7kB ± 0% +2.77% (p=0.016 n=4+5) Read/lz4_524_kB-4 76.5kB ± 0% 78.6kB ± 0% +2.74% (p=0.016 n=5+4) Read/snappy_524_kB-4 39.1kB ± 0% 39.5kB ± 0% +1.19% (p=0.008 n=5+5) Read/flate_524_kB-4 95.4kB ± 0% 97.3kB ± 0% +2.06% (p=0.008 n=5+5) Read/sample_none_66_kB-4 39.9kB ± 0% 41.7kB ± 0% +4.62% (p=0.008 n=5+5) Read/sample_gzip_66_kB-4 686kB ± 0% 715kB ± 0% +4.23% (p=0.016 n=4+5) Read/sample_lz4-64k_66_kB-4 707kB ± 0% 728kB ± 0% +2.88% (p=0.016 n=5+4) Read/sample_lz4-256k_66_kB-4 749kB ± 0% 783kB ± 4% +4.65% (p=0.008 n=5+5) Read/sample_lz4_66_kB-4 748kB ± 0% 770kB ± 0% +2.91% (p=0.016 n=5+4) Read/sample_snappy_66_kB-4 350kB ± 0% 364kB ± 0% +4.04% (p=0.029 n=4+4) Read/sample_flate_66_kB-4 688kB ± 0% 716kB ± 0% +4.08% (p=0.029 n=4+4) Read/sample_none_262_kB-4 10.6kB ± 0% 11.0kB ± 0% +4.23% (p=0.016 n=5+4) Read/sample_gzip_262_kB-4 194kB ± 0% 199kB ± 0% +2.48% (p=0.016 n=4+5) Read/sample_lz4-64k_262_kB-4 165kB ± 0% 169kB ± 0% +2.86% (p=0.016 n=4+5) Read/sample_lz4-256k_262_kB-4 172kB ± 0% 177kB ± 0% +2.92% (p=0.016 n=4+5) Read/sample_lz4-1M_262_kB-4 176kB ± 0% 181kB ± 0% +2.91% (p=0.029 n=4+4) Read/sample_lz4_262_kB-4 176kB ± 0% 181kB ± 0% +2.81% (p=0.016 n=5+4) Read/sample_snappy_262_kB-4 88.1kB ± 0% 90.8kB ± 0% +3.11% (p=0.029 n=4+4) Read/sample_flate_262_kB-4 194kB ± 0% 198kB ± 0% +2.13% (p=0.029 n=4+4) Read/sample_none_524_kB-4 5.56kB ± 0% 5.76kB ± 0% +3.65% (p=0.016 n=5+4) Read/sample_gzip_524_kB-4 95.4kB ± 0% 96.9kB ± 0% +1.56% (p=0.008 n=5+5) Read/sample_lz4-64k_524_kB-4 69.0kB ± 0% 71.0kB ± 0% +2.87% (p=0.008 n=5+5) Read/sample_lz4-256k_524_kB-4 71.8kB ± 0% 73.8kB ± 0% +2.79% (p=0.016 n=4+5) Read/sample_lz4-1M_524_kB-4 74.5kB ± 0% 76.6kB ± 0% +2.75% (p=0.008 n=5+5) Read/sample_lz4_524_kB-4 74.5kB ± 0% 188.4kB ±89% +152.98% (p=0.008 n=5+5) Read/sample_snappy_524_kB-4 40.7kB ± 1% 41.3kB ± 1% +1.58% (p=0.008 n=5+5) Read/sample_flate_524_kB-4 95.2kB ± 0% 96.2kB ± 0% +0.95% (p=0.008 n=5+5) name old allocs/op new allocs/op delta Read/gzip_66_kB-4 13.2k ± 0% 13.2k ± 0% -0.04% (p=0.000 n=5+4) Read/zstd_66_kB-4 18.0k ± 1% 17.8k ± 1% -1.06% (p=0.032 n=5+5) Read/gzip_262_kB-4 4.61k ± 0% 4.61k ± 0% -0.11% (p=0.029 n=4+4) Read/flate_262_kB-4 4.61k ± 0% 4.60k ± 0% -0.11% (p=0.008 n=5+5) Read/zstd_262_kB-4 7.10k ± 2% 6.88k ± 3% -3.01% (p=0.008 n=5+5) Read/gzip_524_kB-4 2.62k ± 0% 2.62k ± 0% -0.06% (p=0.000 n=4+5) Read/zstd_524_kB-4 6.13k ± 2% 5.94k ± 2% -3.03% (p=0.024 n=5+5) Read/sample_flate_66_kB-4 13.2k ± 0% 13.2k ± 0% -0.01% (p=0.029 n=4+4) Read/sample_flate_262_kB-4 4.62k ± 0% 4.61k ± 0% -0.22% (p=0.029 n=4+4) Read/sample_zstd_262_kB-4 7.12k ± 1% 6.97k ± 1% -2.11% (p=0.008 n=5+5) Read/sample_gzip_524_kB-4 2.63k ± 0% 2.63k ± 0% -0.11% (p=0.008 n=5+5) Read/sample_flate_524_kB-4 2.63k ± 0% 2.62k ± 0% -0.15% (p=0.008 n=5+5) Read/sample_zstd_524_kB-4 6.11k ± 2% 5.89k ± 1% -3.68% (p=0.016 n=5+4) ``` Co-authored-by: Owen Diehl <[email protected]>
changhyuni
pushed a commit
to changhyuni/loki
that referenced
this pull request
Nov 8, 2022
**What this PR does / why we need it**: ### BenchmarkWrite Use Go runtime facilities to report MB/s for each operation and compression ratio. Before: ``` BenchmarkWrite/ordered-none-4 1018 1104885 ns/op 1431962 B/op 33 allocs/op BenchmarkWrite/ordered-gzip-4 12 87333201 ns/op 3516699 B/op 696 allocs/op BenchmarkWrite/ordered-lz4-64k-4 51 21483048 ns/op 3095117 B/op 649 allocs/op ``` After: ``` BenchmarkWrite/ordered-none-4 841 1202007 ns/op 1054353.53 MB/s 101.9 %compressed 1569953 B/op 34 allocs/op BenchmarkWrite/ordered-gzip-4 13 86299699 ns/op 3357.25 MB/s 6.891 %compressed 3778896 B/op 702 allocs/op BenchmarkWrite/ordered-lz4-64k-4 49 22332214 ns/op 34107.61 MB/s 9.880 %compressed 3407522 B/op 661 allocs/op ``` Stop collecting all compressed chunks, as this blows up the memory of the benchmark and creates unrealistic test conditions for the later encoding. Before: `4724260 maxresident KB`; after: `375804 maxresident KB` ### BenchmarkRead Replace reporting of MB/s which gave several values for each encoding as Go hunted for the right number of benchmark iterations. Add stats to the context for decoding, otherwise a new stats object is created each time round the loop. Re-order so all unsampled results come before all sampled results. Before: ``` BenchmarkRead/none_66_kB-4 278 4555157 ns/op 131744 B/op 717 allocs/op BenchmarkRead/sample_none_66_kB-4 142 8238861 ns/op 129771 B/op 717 allocs/op BenchmarkRead/gzip_66_kB-4 6 179445187 ns/op 2100498 B/op 15010 allocs/op BenchmarkRead/sample_gzip_66_kB-4 4 261040057 ns/op 2076030 B/op 15028 allocs/op BenchmarkRead/lz4-64k_66_kB-4 19 62855240 ns/op 1722979 B/op 14219 allocs/op BenchmarkRead/sample_lz4-64k_66_kB-4 10 115577796 ns/op 1706040 B/op 14220 allocs/op BenchmarkRead/lz4-256k_66_kB-4 18 66144317 ns/op 1880851 B/op 15078 allocs/op BenchmarkRead/sample_lz4-256k_66_kB-4 9 118405993 ns/op 1800396 B/op 15077 allocs/op BenchmarkRead/lz4-1M_66_kB-4 18 67832189 ns/op 1821806 B/op 15076 allocs/op ... none_524 kB: 1828.92 MB/s none_524 kB: 1809.33 MB/s none_262 kB: 1790.79 MB/s none_66 kB: 1673.84 MB/s ... gzip_66 kB: 647.35 MB/s ... ``` After: ``` BenchmarkRead/none_66_kB-4 286 4377585 ns/op 1725.23 MB/s 41896 B/op 600 allocs/op BenchmarkRead/gzip_66_kB-4 6 180770638 ns/op 655.82 MB/s 723426 B/op 13265 allocs/op BenchmarkRead/lz4-64k_66_kB-4 19 65380524 ns/op 1296.10 MB/s 727571 B/op 12925 allocs/op BenchmarkRead/lz4-256k_66_kB-4 16 63737292 ns/op 1409.75 MB/s 786994 B/op 13707 allocs/op BenchmarkRead/lz4-1M_66_kB-4 18 65963825 ns/op 1362.17 MB/s 770441 B/op 13707 allocs/op BenchmarkRead/lz4_66_kB-4 18 64955415 ns/op 1383.31 MB/s 1003277 B/op 13707 allocs/op BenchmarkRead/snappy_66_kB-4 20 51588959 ns/op 1273.38 MB/s 361872 B/op 5020 allocs/op BenchmarkRead/flate_66_kB-4 6 172147502 ns/op 691.74 MB/s 715706 B/op 13242 allocs/op BenchmarkRead/zstd_66_kB-4 3 359473273 ns/op 390.07 MB/s 427247248 B/op 18551 allocs/op ... BenchmarkRead/sample_none_66_kB-4 135 8533283 ns/op 885.04 MB/s 39866 B/op 600 allocs/op BenchmarkRead/sample_gzip_66_kB-4 5 243016949 ns/op 487.84 MB/s 686358 B/op 13212 allocs/op BenchmarkRead/sample_lz4-64k_66_kB-4 10 114330032 ns/op 741.18 MB/s 707104 B/op 12926 allocs/op BenchmarkRead/sample_lz4-256k_66_kB-4 9 117294928 ns/op 766.05 MB/s 778461 B/op 13709 allocs/op ... ``` Note "MB" in previous code was 2^^20 while Go uses 10^6. **Checklist** - [x] Reviewed the `CONTRIBUTING.md` guide - NA Documentation added - [x] Tests updated - NA `CHANGELOG.md` updated - not user-facing - NA Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`
Abuelodelanada
pushed a commit
to canonical/loki
that referenced
this pull request
Dec 1, 2022
**What this PR does / why we need it**: ### BenchmarkWrite Use Go runtime facilities to report MB/s for each operation and compression ratio. Before: ``` BenchmarkWrite/ordered-none-4 1018 1104885 ns/op 1431962 B/op 33 allocs/op BenchmarkWrite/ordered-gzip-4 12 87333201 ns/op 3516699 B/op 696 allocs/op BenchmarkWrite/ordered-lz4-64k-4 51 21483048 ns/op 3095117 B/op 649 allocs/op ``` After: ``` BenchmarkWrite/ordered-none-4 841 1202007 ns/op 1054353.53 MB/s 101.9 %compressed 1569953 B/op 34 allocs/op BenchmarkWrite/ordered-gzip-4 13 86299699 ns/op 3357.25 MB/s 6.891 %compressed 3778896 B/op 702 allocs/op BenchmarkWrite/ordered-lz4-64k-4 49 22332214 ns/op 34107.61 MB/s 9.880 %compressed 3407522 B/op 661 allocs/op ``` Stop collecting all compressed chunks, as this blows up the memory of the benchmark and creates unrealistic test conditions for the later encoding. Before: `4724260 maxresident KB`; after: `375804 maxresident KB` ### BenchmarkRead Replace reporting of MB/s which gave several values for each encoding as Go hunted for the right number of benchmark iterations. Add stats to the context for decoding, otherwise a new stats object is created each time round the loop. Re-order so all unsampled results come before all sampled results. Before: ``` BenchmarkRead/none_66_kB-4 278 4555157 ns/op 131744 B/op 717 allocs/op BenchmarkRead/sample_none_66_kB-4 142 8238861 ns/op 129771 B/op 717 allocs/op BenchmarkRead/gzip_66_kB-4 6 179445187 ns/op 2100498 B/op 15010 allocs/op BenchmarkRead/sample_gzip_66_kB-4 4 261040057 ns/op 2076030 B/op 15028 allocs/op BenchmarkRead/lz4-64k_66_kB-4 19 62855240 ns/op 1722979 B/op 14219 allocs/op BenchmarkRead/sample_lz4-64k_66_kB-4 10 115577796 ns/op 1706040 B/op 14220 allocs/op BenchmarkRead/lz4-256k_66_kB-4 18 66144317 ns/op 1880851 B/op 15078 allocs/op BenchmarkRead/sample_lz4-256k_66_kB-4 9 118405993 ns/op 1800396 B/op 15077 allocs/op BenchmarkRead/lz4-1M_66_kB-4 18 67832189 ns/op 1821806 B/op 15076 allocs/op ... none_524 kB: 1828.92 MB/s none_524 kB: 1809.33 MB/s none_262 kB: 1790.79 MB/s none_66 kB: 1673.84 MB/s ... gzip_66 kB: 647.35 MB/s ... ``` After: ``` BenchmarkRead/none_66_kB-4 286 4377585 ns/op 1725.23 MB/s 41896 B/op 600 allocs/op BenchmarkRead/gzip_66_kB-4 6 180770638 ns/op 655.82 MB/s 723426 B/op 13265 allocs/op BenchmarkRead/lz4-64k_66_kB-4 19 65380524 ns/op 1296.10 MB/s 727571 B/op 12925 allocs/op BenchmarkRead/lz4-256k_66_kB-4 16 63737292 ns/op 1409.75 MB/s 786994 B/op 13707 allocs/op BenchmarkRead/lz4-1M_66_kB-4 18 65963825 ns/op 1362.17 MB/s 770441 B/op 13707 allocs/op BenchmarkRead/lz4_66_kB-4 18 64955415 ns/op 1383.31 MB/s 1003277 B/op 13707 allocs/op BenchmarkRead/snappy_66_kB-4 20 51588959 ns/op 1273.38 MB/s 361872 B/op 5020 allocs/op BenchmarkRead/flate_66_kB-4 6 172147502 ns/op 691.74 MB/s 715706 B/op 13242 allocs/op BenchmarkRead/zstd_66_kB-4 3 359473273 ns/op 390.07 MB/s 427247248 B/op 18551 allocs/op ... BenchmarkRead/sample_none_66_kB-4 135 8533283 ns/op 885.04 MB/s 39866 B/op 600 allocs/op BenchmarkRead/sample_gzip_66_kB-4 5 243016949 ns/op 487.84 MB/s 686358 B/op 13212 allocs/op BenchmarkRead/sample_lz4-64k_66_kB-4 10 114330032 ns/op 741.18 MB/s 707104 B/op 12926 allocs/op BenchmarkRead/sample_lz4-256k_66_kB-4 9 117294928 ns/op 766.05 MB/s 778461 B/op 13709 allocs/op ... ``` Note "MB" in previous code was 2^^20 while Go uses 10^6. **Checklist** - [x] Reviewed the `CONTRIBUTING.md` guide - NA Documentation added - [x] Tests updated - NA `CHANGELOG.md` updated - not user-facing - NA Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md`
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What this PR does / why we need it:
BenchmarkWrite
Use Go runtime facilities to report MB/s for each operation and compression ratio.
Before:
After:
Stop collecting all compressed chunks, as this blows up the memory of the benchmark and creates unrealistic test conditions for the later encoding.
Before:
4724260 maxresident KB; after:375804 maxresident KBBenchmarkRead
Replace reporting of MB/s which gave several values for each encoding as Go hunted for the right number of benchmark iterations.
Add stats to the context for decoding, otherwise a new stats object is created each time round the loop.
Re-order so all unsampled results come before all sampled results.
Before:
After:
Note "MB" in previous code was 2^^20 while Go uses 10^6.
Checklist
CONTRIBUTING.mdguideCHANGELOG.mdupdated - not user-facingdocs/sources/upgrading/_index.md