Go并行gzip压缩:pgzip

jopen 10年前

Go并行gzip压缩。它完全兼容gzip可直接替换 "compress/gzip"。它将待压缩内容拆分成多块然后并行压缩,对于大数据量压缩非常好用。

var b bytes.Buffer  w := gzip.NewWriter(&b)  w.SetConcurrency(100000, 10)  w.Write([]byte("hello, world\n"))  w.Close()

性能比较

Compression cost is usually about 0.2% with default settings with a block size of 250k.

Example with GOMAXPROC set to 4 (dual core with 2 hyperthreads)

Compressor MB/sec speedup size size overhead
gzip (golang) 15.082MB/s 1.0x 6.405.193 0%
pgzip (golang) 26.736MB/s 1.8x 6.421.585 0.2%
bgzf (golang) 29.525MB/s 1.9x 6.875.913 7.3%


项目主页:http://www.open-open.com/lib/view/home/1413516367512