[OmniOS-discuss] A problem and puzzle with disappearing ZFS snapshots

[Chris Siebenmann]

 I wrote, about our mysteriously disappearing snapshots:
>  [...] For example, is there some way where snapshots can be removed
> without that being logged in 'zpool history'?

 The answer to my question turns out to be 'yes'. If you do:
	rmdir <fs>/.zfs/snapshot/<snapshot-name>

and ZFS accepts this, there is no entry made in 'zpool history'; the
snapshot is silently deleted.

 A successful rmdir of a snapshot can be done by root either locally or
over NFS, if the NFS client has been given root permissions. Like other
snapshot removals, this rmdir will be blocked if the snapshot has a hold
on it ('zfs hold'/'zfs release').

 We have some client systems that have NFS root access to this filesystem.
Notably, one of them is a Samba server, and it is possible that some Samba
client is doing something that persuades the Samba server to issue a rmdir
against .zfs/snapshot/<snapname> as root, possibly for all things in the
directory that it sees at the time.

 I guess this is where I reach for DTrace to see if we can trace usage
of this snapshot removal path to see if we can confirm that something
is removing (or attempting to remove[*]) snapshots through this path.

	- cks
[*: I have just changed our scripting to apply holds to all of these
    automatically created snapshots, and remove them just before the
    script would normally delete the snapshot.
]