[OmniOS-discuss] multithreaded gzip (or equivalent) and moving some files while preserving file trees

[Valrhona]

Thanks for the tip. If I understand correctly, lzop is basically a
drop-in replacement for gzip that is much faster, but single-threaded.
Is there a comparison anywhere between (single-threaded) lzop and
(multithreaded) pigz or pbzip2? I have a hex-core xeon and 72 GB of
RAM so I expect the compression to be IO-bound rather than CPU-bound,
but that is just a guess. And for ZFS streams, is there a reason to
prefer any one of these programs over another? Thanks!

On Fri, Aug 16, 2013 at 4:46 PM, Bob Friesenhahn
<bfriesen at simple.dallas.tx.us> wrote:
> On Thu, 15 Aug 2013, Valrhona wrote:
>
>> Thanks to OmniTI for making a fantastic product for the community!
>> I am doing a bunch of backups, and trying to organize data, and have two
>> questions:
>>
>> 1. Is there a better alternative, perhaps in the new package repositories,
>> for gzip-style compression that is multithreaded? I am doing the usualy zfs
>> send to a
>> file, which I then backup to tape. Using gzip makes the process like 100x
>> slower than if I just dump the zfs stream to an uncompressed file, and so
>> it's not
>> practical from a time standpoint.
>
>
> Use 'lzop' with default compression settings instead.  It is much more CPU
> efficient than gzip.  The compression ratio is somewhat less than gzip but
> the compression rate is vastly better.
>
> Bob
> --
> Bob Friesenhahn
> bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
> GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/