[OmniOS-discuss] offline dedup

[Nikolam]

On 1/5/15, Rune Tipsmark <rt at steait.net> wrote:
> hi all,
>
> does anyone know if offline dedup is something we can expect in the future
> of ZFS?
>
> I have some backup boxes with 50+TB on them and only 32GB Ram and even zdb
> -S crashes due to lack of memory. Seems complete overkill to put 256+GB ram
> in a slow backup box... and if I enable dedup as is, it will crash after
> writing a few TB - reboot required.

As I understand, RAM needed for deduplicated data in use relates to
how much data is inside dataset that is deduplicated.
So problem with large datasets is that they need large amount of RAM
reserved during use IF deduplication is turned on.

Offline deduplication would be copying one zfs dataset that is not
deduplicated to another dataset that has deduplication turned on.
Amount of RAM required by deduplicated dataset would be required all
the time uring the use of dataset with deduplication turned on.
But if deduplication is turned off on new dataset after copying, then
that dataset would not use additional RAM during regular use.

So, maybe one could make snapshot after creating new deduplicated
dataset and then turn deduplication off.
Then you do your everyday doing with dedup off.
After end of a day, you make a snapshot of a new state of dataset,
you make clone of a old snapshot state of dataset , turn ON
deduplication on cloned dataset and do zfs send between old and new
snapshot.
Then you turn off deduplication on new state of dataset, snapshot it,
clone it again and then unmount existing working dataset and remount
new one. (and possibly destroying old one that is not deduplicated,
while retaining previous snapshot  before last deduplication)

That way you have your offline deduplication, if you write script for it
and you make sure that you have enough RAM for deduplication of newly
added data during some period, to take small enough time to be called
"offline" and finish during off-hours.

Initial deduplication of existing data in dataset could need a lot of
RAM if there is many data in it.
So one might consider moving data to several smaller datasets and
deduplicate them separately.

Anyway it needs testing and deduplication generally needs RAM relative
to data deduplicated.
It helps if what I described above is done regularly, while Pool is filled up.

Maybe one can do full reliable backup (one do backups, zfs and
replication is no substitution for separate physical backups) And the
fill pool like described in iterations, if you are sure you will use
that pool with small amount of RAM+scripted offline deduplications.

Only problem is that even scripted ("offline") deduplications need
large amount of RAM the moment deduplication is turned ON on large
dataset, so even offline deduplication on large dataset would be a
problem on small memory systems.



MAYBE if one accept slowdown of using SSD(or SSDs) or even space on
spindle disks for deduplication data, that should originally be inside
RAM.
That would require really investing in development and writing
deduplication script and extending ZFS to use on-disk/on-SSD space for
Deduplication info.

It is only a question how much slower would be that kind of
deduplication on large datasets, comparing to in-RAM data and would it
finish on time to be called "offline" during low-memory machine "off
hours".

As rule of the thumb, one needs RAM if ever wants to use deduplication
and it works nice if so.
If you don't have enough RAM, just forget on deduplication on large
datasets or buy more RAM.
One can be careful using snapshots to laverage CoW without dedulication.

Or invest Development Money and development hours
to extend ZFS to be able to use SSDs or disks for deduplication space
during offline dedulication.
Maybe such feature could be called
"Copy on write cp" and used on scripts for offline deduplication,
instead of turning deduplication ON and OFF.