[OmniOS-discuss] A problem and puzzle with disappearing ZFS snapshots

[Chris Siebenmann]

 For the interest of bystanders, here's the resolution to our puzzle.

I wrote:
>  I wrote, about our mysteriously disappearing snapshots:
> >  [...] For example, is there some way where snapshots can be removed
> > without that being logged in 'zpool history'?
> 
>  The answer to my question turns out to be 'yes'. If you do:
> 	rmdir <fs>/.zfs/snapshot/<snapshot-name>
> 
> and ZFS accepts this, there is no entry made in 'zpool history'; the
> snapshot is silently deleted.

 This is actually wrong, because it turns out that I hadn't read the
zpool manpage about 'zpool history' carefully enough. By default 'zpool
history' shows only ZFS commands; however, you can get it to show
'internally logged ZFS events' in addition with the -i switch, and this
did show our snapshot deletions.

('zpool history -i' has some interesting output around scrubs and
resilvers too.)

>  A successful rmdir of a snapshot can be done by root either locally
> or over NFS, if the NFS client has been given root permissions. Like
> other snapshot removals, this rmdir will be blocked if the snapshot
> has a hold on it ('zfs hold'/'zfs release').
>
>  We have some client systems that have NFS root access to this
> filesystem.  Notably, one of them is a Samba server, and it is
> possible that some Samba client is doing something that persuades the
> Samba server to issue a rmdir against .zfs/snapshot/<snapname> as
> root, possibly for all things in the directory that it sees at the
> time.

 It turns out that I was unjustly suspicious of Samba. The real culprit
was part of our home-grown NFS mount management system, which will
unmount NFS filesystems that it thinks are not supposed to be there and
as part of doing this will rmdir their mount point directories. When
it was dealing with a snapshot NFS mount, this rmdir naturally deleted
the snapshot (well, when done from our Samba server, with its root
permissions). Snapshots were being accessed on our Samba server for the
straightforward reason that some users were looking in them for deleted
files.

(Linux materializes synthetic NFS mounts for .zfs/snapshot/<snapshot>
filesystems when they're accessed on a NFS client. These are almost
undistinguishable from regular NFS mounts, so our NFS mount management
system saw them as extra, not-supposed-to-be-there NFS mounts that it
should deal with.)

 I'd like to thank Dan McDonald and the others here for their help
in leading me towards the ultimate solution.

	- cks