[OmniOS-discuss] dedup causes zfs/omnios to drop connections.

[Jason Cox]

I have found that turning on compression gets more savings when storing my
backups then I got with dedup on. I run gzip-9 for my backup dataset. I
think dedup is good if you are storing a lot of files that really are the
same. Your mileage may vary compared to mine. I do a full backup once a
week and incrementals daily.

On Mon, Dec 15, 2014 at 2:17 PM, Rune Tipsmark <rt at steait.net> wrote:
>
> we have only 24GB of ram on this system...
> I was under the impression it would not require much when the block size
> was larger, we are on 64kb, so would expect around 2GB per 1TB.
> yet we cannot even get more than a few TB on the system before it dies.
>
> the main purpose with this system was dedup, no problem with slow speed...
> its for backup only, so could not care less if its rather slow - but not
> responding is not acceptable.
> br,
> Rune
> ________________________________________
> From: OmniOS-discuss <omnios-discuss-bounces at lists.omniti.com> on behalf
> of Dominik Hassler <hasslerd at gmx.li>
> Sent: Monday, December 15, 2014 10:11 PM
> To: omnios-discuss at lists.omniti.com
> Subject: Re: [OmniOS-discuss] dedup causes zfs/omnios to drop connections.
>
> Hi,
>
> we used dedup on a production machine w/ 256 GB RAM, but disabled it
> after a couple of days due to huge performance impact.
>
> I would not recommend to use dedup even when having "enough" RAM.
>
> On 12/15/2014 09:53 PM, Dan McDonald wrote:
> >
> >> On Dec 15, 2014, at 3:43 PM, Rune Tipsmark <rt at steait.net> wrote:
> >>
> >> hi all,
> >>
> >> got a new system I was intending on using as backup repository.
> Whenever dedup is enabled it dies after anywhere between 5 and 30 minutes.
> I need to reboot OmniOS to get it back online.
> >> the files being copied onto the zfs vols are rather large, about ~2TB
> each... if I copy smaller files, say 400GB or so, it takes longer for it to
> crash.
> >>
> >> what can be done to fix this? after Windows (initiator) looses the
> connection (both Fibre Channel and iSCSI) I still see a lot of disk
> activity using iostat - disks remain active for minutes after the copying
> has died... its like ZFS cannot handle dedup of large files..
> >
> > Dedup is a memory pig and not very well implemented in ZFS.  I'd highly
> recommend against it in production.  Either that, or really increase your
> memory for your system in question.  There was some work going on at
> Nexenta to perhaps put the dedup tables (DDTs) onto a dedicated slog-like
> device, but I believe that work stalled.
> >
> > Sorry,
> > Dan
> >
> >
> > _______________________________________________
> > OmniOS-discuss mailing list
> > OmniOS-discuss at lists.omniti.com
> > http://lists.omniti.com/mailman/listinfo/omnios-discuss
> >
> _______________________________________________
> OmniOS-discuss mailing list
> OmniOS-discuss at lists.omniti.com
> http://lists.omniti.com/mailman/listinfo/omnios-discuss
>


-- 
Jason Cox
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://omniosce.org/ml-archive/attachments/20141216/7beae212/attachment-0001.html>