[OmniOS-discuss] dedup causes zfs/omnios to drop connections.

[Rune Tipsmark]

we have only 24GB of ram on this system...
I was under the impression it would not require much when the block size was larger, we are on 64kb, so would expect around 2GB per 1TB.
yet we cannot even get more than a few TB on the system before it dies. 

the main purpose with this system was dedup, no problem with slow speed... its for backup only, so could not care less if its rather slow - but not responding is not acceptable.
br,
Rune
________________________________________
From: OmniOS-discuss <omnios-discuss-bounces at lists.omniti.com> on behalf of Dominik Hassler <hasslerd at gmx.li>
Sent: Monday, December 15, 2014 10:11 PM
To: omnios-discuss at lists.omniti.com
Subject: Re: [OmniOS-discuss] dedup causes zfs/omnios to drop connections.

Hi,

we used dedup on a production machine w/ 256 GB RAM, but disabled it
after a couple of days due to huge performance impact.

I would not recommend to use dedup even when having "enough" RAM.

On 12/15/2014 09:53 PM, Dan McDonald wrote:
>
>> On Dec 15, 2014, at 3:43 PM, Rune Tipsmark <rt at steait.net> wrote:
>>
>> hi all,
>>
>> got a new system I was intending on using as backup repository. Whenever dedup is enabled it dies after anywhere between 5 and 30 minutes. I need to reboot OmniOS to get it back online.
>> the files being copied onto the zfs vols are rather large, about ~2TB each... if I copy smaller files, say 400GB or so, it takes longer for it to crash.
>>
>> what can be done to fix this? after Windows (initiator) looses the connection (both Fibre Channel and iSCSI) I still see a lot of disk activity using iostat - disks remain active for minutes after the copying has died... its like ZFS cannot handle dedup of large files..
>
> Dedup is a memory pig and not very well implemented in ZFS.  I'd highly recommend against it in production.  Either that, or really increase your memory for your system in question.  There was some work going on at Nexenta to perhaps put the dedup tables (DDTs) onto a dedicated slog-like device, but I believe that work stalled.
>
> Sorry,
> Dan
>
>
> _______________________________________________
> OmniOS-discuss mailing list
> OmniOS-discuss at lists.omniti.com
> http://lists.omniti.com/mailman/listinfo/omnios-discuss
>