[OmniOS-discuss] NFS Datastore vmware esxi failover

[Saso Kiselkov]

On 11/8/13, 6:08 PM, Matt Weiss wrote:
> 
> On 11/8/2013 11:36 AM, Saso Kiselkov wrote:
>> On 11/8/13, 4:17 PM, Matt Weiss wrote:
>>> I am working on a failover script using OmniOS as a NFS server.
>>>
>>> According to VMware, if I mount and nfs datastore via its IP Address
>>> then I should be able to move the IP around and still mount it, however
>>> it is not working right.
>>>
>>> For example:
>>>
>>> On an ESXi instance (5.1U1) I mount the following NFS Datastore
>>> 172.16.50.100
>>> /tank/vmrep
>>> which amounts to a UUID of 6c0c1d0d-928ef591 in /vmfs/volumes
>>>
>>>
>>> omni-rep1: 172.16.50.1
>>> omni-rep2: 172.16.50.2
>>>
>>> I am using zrep to failover my zfs dataset.
>>> http://www.bolthole.com/solaris/zrep/zrep.documentation.html
>>>
>>> Essential, it puts primary into read-only, does a zfs send/receive, then
>>> sets the secondary to rw.
>>>
>>>
>>> To expose my dataset (tank/vmrep) I am using sharenfs property of zfs. I
>>> have created a virtual ip to use for this purpose.
>>>
>>> #setnfsip.sh
>>> ipadm create-addr -T static -a 172.16.50.100/24 vmxnet3s0/nfs
>>>
>>> #removenfsip.sh
>>> ipadm delete-addr vmxnet3s0/nfs
>>>
>>>
>>> So, when I want to failover, I just do the following:
>>>
>>> #!/bin/sh
>>> #zfs unshare tank/vmrep
>>> #sleep 5
>>> /scripts/removenfsip.sh
>>> sync
>>> sleep 5
>>> #zrep sync tank/vmrep
>>> #sleep 5
>>> #the following does the zfs snapshot/send/receive
>>> zrep failover tank/vmrep
>>> sleep 5
>>> #ssh 172.16.50.2 /usr/sbin/zfs share tank/vmrep
>>> #sleep 5
>>> ssh 172.16.50.2 /scripts/setnfsip.sh
>>>
>>>
>>> So, all goes well, omni-rep2 is now exporting tank/vmrep with NFS, it
>>> has the 172.16.50.100 ip address, it is the exact replica of omni-rep1.
>>>
>>> The problem is in ESXi the datastore goes inaccessable.  I can fail back
>>> and the datastore comes back online like fine.  I can mount the nfs
>>> datastore as a new one with the .100 ip on omni-rep2 so it is being
>>> exported properly.
>>>
>>> According to the last paragraph of this
>>>
>>> https://communities.netapp.com/community/netapp-blogs/getvirtical/blog/2011/09/28/nfs-datastore-uuids-how-they-work-and-what-changed-in-vsphere-5
>>>
>>>
>>>
>>> It should work, I have merely changed which host is broadcasting my
>>> datastore's IP address.
>>>
>>> I know a guy named saso? did some iScsi failover recently and noted it
>>> worked with NFS.  I am just wondering what I am missing here.
>> I haven't done NFS datastore failover from ESXi myself, but off the top
>> of my head I guess what's going haywire here is that you're setting the
>> dataset read-only before moving it over. Don't do that. Simply tear down
>> the IP address, migrate the dataset, set up a new NFS share on the
>> target machine and then reinstate the IP address at the target. ESXi
>> aggressively monitors the health of its datastores and if it gets to a
>> state it can't deal with (e.g. write a datastore that refuses to process
>> it), it will offline the whole datastore, awaiting administrator
>> intervention.
>>
>> Don't worry about the datastore being offline for a while, ESXi will
>> hold VM writes and the VMs themselves won't usually complain for up to
>> 1-2 minutes (defaults on Windows/Linux).
>>
> I have experimented with
> sharing / unsharing
> leaving shared all the time
> setting RO, RW
> changing IP before/after etc
> adding sleeps, syncs
> 
> So far all have the same result.  I believe the issue now lies with the
> NFS state file.  I may be able to get away with migrating that file with
> my failover script, but we will see.
> 
> Does anyone know how to change OmniOS NFS server parameters?  I would be
> wanting to put the state files into the zfs dataset somewhere to perhaps
> handle that.  Somehow I believe pacemaker takes care of this, so surely,
> somehow NFS can be made to be a true failover.

Have you tried:

1) tear down IP address on machine A
2) migrate dataset to machine B
3) share dataset on machine B
4) set up IP address on machine B

In this fashion you guarantee that at any time when the IP address is
visible, the datastore is visible from it as well - this is crucial for
ESX. If at any point it connects to your NFS server, but the datastore
is not available, then it will declare the datastore unavailable and
back off. The same thing happens with iSCSI targets.

-- 
Saso