[OmniOS-discuss] Clues for tracking down a drastic ZFS fs space difference?

[Chris Siebenmann]

 We have a filesystem/dataset with no snapshots, no subordinate
filesystems, nothing complicated (and no compression), that has a
drastic difference in space used between what df/zfs list/etc report at
the ZFS level and what du reports at the filesystem level. ZFS says

NAME              PROPERTY       VALUE   SOURCE
fs0-core-01/cs/8  used           70.5G   -
fs0-core-01/cs/8  usedbydataset  70.5G   -

On the other hand, 'du -h' says 17 GB, which is what we'd expect.
More alarmingly, this dataset seems to keep steadily growing at the
ZFS level despite 'du -h' figures staying constant or even going down.
On April 2nd it was reporting a 'du -h' of 22 GB but 48 GB used at the
ZFS level; as you can see, it's added 22 GB of ZFS usage in less than
a month while losing 5 GB at the user level.

 What sort of things should I be looking at to try to figure out why
this is happening, including with eg zdb? Are there any obvious reasons
why this would be happening? Is there any easy way to fix this short of
'copy all data to a new dataset, destroy old dataset, put new dataset
in the place of the old?'

(The entire pool is on mirrored vdevs and has ashift=12. And looking at
it two other datasets in this pool have similar issues, although neither
are as drastically divergent; one is 360 GB vs 302 GB, another is 28.8
GB vs 20 GB.)

 Thanks in advance.

	- cks