Skip to content

Task List

Jump to bottom
Joe Tsai edited this page Feb 19, 2016 · 26 revisions

Current high-level objectives

By popular request, Brotli is currently the main focus. However, in order to satisfy that objective, a DEFLATE implementation will be done first.

  • Implement flate.Writer
  • Implement brotli.Writer

The Brotli format shares many similarities with DEFLATE. At its core, it is built on the ideas of LZ77 and Huffman bit-encoding. Thus, implementing DEFLATE first will allow for a better understanding of how to implement Brotli well. Given that the Readers were able to share much code, it is likely that the Writers will also be able to share much code.

Items to work on by package

  • brotli
    • Reader
      • Make use of internal/prefix
      • Add more tests for malicious and strange inputs
    • Writer
      • Implement me!
  • bzip2
    • Reader
      • Handle CRC32 checksums at the block and stream level
      • Handle multiple back-to-back files
      • Add tests for malicious and strange inputs
      • (performance) Reduce trash generation
      • (performance) Add chunking between stages to reduce total memory bandwidth
      • (performance) Investigate other ways to improve performance
    • Writer
      • Optimize assignment of prefix trees for better compression ratio
      • Investigate why internal/sais is crashing on some inputs
      • Investigate various BWT construction methods:
        • bwtis, sais, qsufsort, divsufsort
      • (performance) Remove trash generation
      • (performance) Add chunking between stages to reduce total memory bandwidth
      • (performance) Investigate other ways to improve performance
      • Investigate if blocks can easily be byte aligned
  • flate
    • Reader
      • Make use of internal/dict when available
    • Writer
      • Implement me!
  • xz
    • Implement me!
  • internal/prefix
  • internal/dict
    • Implement a generic LZ77 dictionary for use in brotli and flate
  • internal/cgo
    • Move C wrappers here
    • Make C wrappers correct (they violate Go1.6 pointer passing rules)
  • internal/tool/bench
    • Investigate why Brotli tests fail
  • internal/tool/fuzzer
    • Test round-trip Go->Go functionality
    • Test round-trip C->Go functionality
    • Test round-trip Go->C functionality
Clone this wiki locally