[OmniOS-discuss] System hangs randomly
Hafiz Rafibeyli
rafibeyli at gmail.com
Mon Nov 11 07:45:28 UTC 2013
Helllo,
Jim and Richard thank you for your quick answers.
I will control and change controller mode to AHCI,but I think Richard right.
My OS disks are new,but I have some suspicions on Intel X-25E series.
regards
Hafiz.
----- Orijinal Mesaj -----
Kimden: omnios-discuss-request at lists.omniti.com
Kime: omnios-discuss at lists.omniti.com
Gönderilenler: 9 Kasım Cumartesi 2013 0:02:32
Konu: OmniOS-discuss Digest, Vol 20, Issue 7
Send OmniOS-discuss mailing list submissions to
omnios-discuss at lists.omniti.com
To subscribe or unsubscribe via the World Wide Web, visit
http://lists.omniti.com/mailman/listinfo/omnios-discuss
or, via email, send a message with subject or body 'help' to
omnios-discuss-request at lists.omniti.com
You can reach the person managing the list at
omnios-discuss-owner at lists.omniti.com
When replying, please edit your Subject line so it is more specific
than "Re: Contents of OmniOS-discuss digest..."
Today's Topics:
1. Re: NFS Datastore vmware esxi failover (Matt Weiss)
2. Re: NFS Datastore vmware esxi failover (Richard Elling)
3. Re: system hangs randomly (Richard Elling)
----------------------------------------------------------------------
Message: 1
Date: Fri, 08 Nov 2013 12:08:42 -0600
From: Matt Weiss <mweiss at cimlbr.com>
To: omnios-discuss at lists.omniti.com
Subject: Re: [OmniOS-discuss] NFS Datastore vmware esxi failover
Message-ID: <527D28AA.7000509 at cimlbr.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
On 11/8/2013 11:36 AM, Saso Kiselkov wrote:
> On 11/8/13, 4:17 PM, Matt Weiss wrote:
>> I am working on a failover script using OmniOS as a NFS server.
>>
>> According to VMware, if I mount and nfs datastore via its IP Address
>> then I should be able to move the IP around and still mount it, however
>> it is not working right.
>>
>> For example:
>>
>> On an ESXi instance (5.1U1) I mount the following NFS Datastore
>> 172.16.50.100
>> /tank/vmrep
>> which amounts to a UUID of 6c0c1d0d-928ef591 in /vmfs/volumes
>>
>>
>> omni-rep1: 172.16.50.1
>> omni-rep2: 172.16.50.2
>>
>> I am using zrep to failover my zfs dataset.
>> http://www.bolthole.com/solaris/zrep/zrep.documentation.html
>>
>> Essential, it puts primary into read-only, does a zfs send/receive, then
>> sets the secondary to rw.
>>
>>
>> To expose my dataset (tank/vmrep) I am using sharenfs property of zfs. I
>> have created a virtual ip to use for this purpose.
>>
>> #setnfsip.sh
>> ipadm create-addr -T static -a 172.16.50.100/24 vmxnet3s0/nfs
>>
>> #removenfsip.sh
>> ipadm delete-addr vmxnet3s0/nfs
>>
>>
>> So, when I want to failover, I just do the following:
>>
>> #!/bin/sh
>> #zfs unshare tank/vmrep
>> #sleep 5
>> /scripts/removenfsip.sh
>> sync
>> sleep 5
>> #zrep sync tank/vmrep
>> #sleep 5
>> #the following does the zfs snapshot/send/receive
>> zrep failover tank/vmrep
>> sleep 5
>> #ssh 172.16.50.2 /usr/sbin/zfs share tank/vmrep
>> #sleep 5
>> ssh 172.16.50.2 /scripts/setnfsip.sh
>>
>>
>> So, all goes well, omni-rep2 is now exporting tank/vmrep with NFS, it
>> has the 172.16.50.100 ip address, it is the exact replica of omni-rep1.
>>
>> The problem is in ESXi the datastore goes inaccessable. I can fail back
>> and the datastore comes back online like fine. I can mount the nfs
>> datastore as a new one with the .100 ip on omni-rep2 so it is being
>> exported properly.
>>
>> According to the last paragraph of this
>>
>> https://communities.netapp.com/community/netapp-blogs/getvirtical/blog/2011/09/28/nfs-datastore-uuids-how-they-work-and-what-changed-in-vsphere-5
>>
>>
>> It should work, I have merely changed which host is broadcasting my
>> datastore's IP address.
>>
>> I know a guy named saso? did some iScsi failover recently and noted it
>> worked with NFS. I am just wondering what I am missing here.
> I haven't done NFS datastore failover from ESXi myself, but off the top
> of my head I guess what's going haywire here is that you're setting the
> dataset read-only before moving it over. Don't do that. Simply tear down
> the IP address, migrate the dataset, set up a new NFS share on the
> target machine and then reinstate the IP address at the target. ESXi
> aggressively monitors the health of its datastores and if it gets to a
> state it can't deal with (e.g. write a datastore that refuses to process
> it), it will offline the whole datastore, awaiting administrator
> intervention.
>
> Don't worry about the datastore being offline for a while, ESXi will
> hold VM writes and the VMs themselves won't usually complain for up to
> 1-2 minutes (defaults on Windows/Linux).
>
I have experimented with
sharing / unsharing
leaving shared all the time
setting RO, RW
changing IP before/after etc
adding sleeps, syncs
So far all have the same result. I believe the issue now lies with the
NFS state file. I may be able to get away with migrating that file with
my failover script, but we will see.
Does anyone know how to change OmniOS NFS server parameters? I would be
wanting to put the state files into the zfs dataset somewhere to perhaps
handle that. Somehow I believe pacemaker takes care of this, so surely,
somehow NFS can be made to be a true failover.
------------------------------
Message: 2
Date: Fri, 8 Nov 2013 13:54:50 -0800
From: Richard Elling <richard.elling at richardelling.com>
To: Matt Weiss <mweiss at cimlbr.com>
Cc: omnios-discuss <omnios-discuss at lists.omniti.com>
Subject: Re: [OmniOS-discuss] NFS Datastore vmware esxi failover
Message-ID: <3BA1C366-1A6B-448B-B299-9264183B9997 at RichardElling.com>
Content-Type: text/plain; charset="us-ascii"
Comment below...
On Nov 8, 2013, at 8:17 AM, Matt Weiss <mweiss at cimlbr.com> wrote:
> I am working on a failover script using OmniOS as a NFS server.
>
> According to VMware, if I mount and nfs datastore via its IP Address then I should be able to move the IP around and still mount it, however it is not working right.
>
> For example:
>
> On an ESXi instance (5.1U1) I mount the following NFS Datastore
> 172.16.50.100
> /tank/vmrep
> which amounts to a UUID of 6c0c1d0d-928ef591 in /vmfs/volumes
>
>
> omni-rep1: 172.16.50.1
> omni-rep2: 172.16.50.2
>
> I am using zrep to failover my zfs dataset.
> http://www.bolthole.com/solaris/zrep/zrep.documentation.html
>
> Essential, it puts primary into read-only, does a zfs send/receive, then sets the secondary to rw.
>
>
> To expose my dataset (tank/vmrep) I am using sharenfs property of zfs. I have created a virtual ip to use for this purpose.
>
> #setnfsip.sh
> ipadm create-addr -T static -a 172.16.50.100/24 vmxnet3s0/nfs
>
> #removenfsip.sh
> ipadm delete-addr vmxnet3s0/nfs
>
>
> So, when I want to failover, I just do the following:
>
> #!/bin/sh
> #zfs unshare tank/vmrep
> #sleep 5
> /scripts/removenfsip.sh
> sync
> sleep 5
> #zrep sync tank/vmrep
> #sleep 5
> #the following does the zfs snapshot/send/receive
> zrep failover tank/vmrep
> sleep 5
> #ssh 172.16.50.2 /usr/sbin/zfs share tank/vmrep
> #sleep 5
> ssh 172.16.50.2 /scripts/setnfsip.sh
>
>
> So, all goes well, omni-rep2 is now exporting tank/vmrep with NFS, it has the 172.16.50.100 ip address, it is the exact replica of omni-rep1.
It is a replica, but it isn't the same from an NFS perspective. The files may have the
same contents, but the NFSv3 file handles are different because they are in two different
file systems. For this to work, the clients would have to remount, which blows your
requirement for transparency.
>
> The problem is in ESXi the datastore goes inaccessable. I can fail back and the datastore comes back online like fine. I can mount the nfs datastore as a new one with the .100 ip on omni-rep2 so it is being exported properly.
Yes, works as designed.
>
> According to the last paragraph of this
>
> https://communities.netapp.com/community/netapp-blogs/getvirtical/blog/2011/09/28/nfs-datastore-uuids-how-they-work-and-what-changed-in-vsphere-5
>
> It should work, I have merely changed which host is broadcasting my datastore's IP address.
>
> I know a guy named saso? did some iScsi failover recently and noted it worked with NFS. I am just wondering what I am missing here.
Saso's solution, and indeed most failover cluster solutions, use shared storage and not
ZFS replication between the nodes. One good reason to do this is so you can transparently
failover NFS service.
NB, the replication method is used often for disaster recovery, because in DR there often is
no transparent failover requirement.
-- richard
--
Richard.Elling at RichardElling.com
+1-760-896-4422
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://omniosce.org/ml-archive/attachments/20131108/e9b7d40b/attachment-0001.html>
------------------------------
Message: 3
Date: Fri, 8 Nov 2013 14:02:26 -0800
From: Richard Elling <richard.elling at richardelling.com>
To: jimklimov at cos.ru
Cc: omnios-discuss at lists.omniti.com
Subject: Re: [OmniOS-discuss] system hangs randomly
Message-ID: <DB663170-F9B6-472D-BC6A-9DB6D2A4F1C5 at RichardElling.com>
Content-Type: text/plain; charset="us-ascii"
On Nov 8, 2013, at 8:20 AM, Jim Klimov <jimklimov at cos.ru> wrote:
> The logs specify that your IDE devices (I believe, these are the rpool
> SSDs in legacy mode) return errors on reads and timeout on retries or
> resets. This may mean a few things:
>
> 1) Imminent device death i.e. due to wear over lifetime, try to get
> these replaced with new units (especially if their age or some actual
> diagnostics results from "smartctl" or vendor tools also indicate the
> possibility of such scenario)
I vote for this one. The X-25E are well-known for behaving this way as a
failure mode. The only recourse is to replace the disk.
>
> 2) Bad diagnostics, perhaps due to IDE protocol limitations - try to
> switch the controller into SATA mode and use some illumos live media
> (OI LiveCD/LiveUSB or OmniOS equivalents) to boot the server with the
> rpool disks in SATA mode and run:
This isn't the cause or solution for the disk's woes, but I recommend going
to AHCI mode at your convenience. You might be able to replace the disk
without an outage, but this step will require an outage.
-- richard
>
> zpool import -N -R /a -f rpool
> mount -F zfs rpool/ROOT/your_BE_name /a && \
> touch /a/reconfigure
> zpool export rpool
>
> Depending on your OS setup, the BE mounting may require some other
> command (like "zfs mount rpool/ROOT/your_BE_name").
>
> This routine mounts the pool, indicates to the BE that it should make
> new device nodes (so it runs "devfsadm" early in the boot), and exports
> the pool. In the process, the rpool ZFS labels begin referencing the new
> hard-disk device node names which is what the rootfs procedure relies
> on. In some more difficult cases it might help to also copy (rsync) the
> /dev/ and /devices/ from the live environment into the on-disk BE so
> that these device names saved into the pool labels would match those
> discovered by the kernel upon boot.
>
> Do have backups; it might make sense to complete this experiment with
> one of the mirror halves removed, so that if nothing works (even rolling
> back to an IDE-only setup) you can destroy this half's content and boot
> in IDE mode from the other half and re-attach the mirrored part to it.
>
> As a variant, it might make sense (if you'd also refresh the hardware)
> to attach the new device(s) to the rpool as a 3/4-way mirror, and then
> completing the switcheroo to SATA with only the new couple plugged in -
> you'd be able to fall back on the old and tested set if all goes wrong
> somehow.
>
> Good luck,
> //Jim
>
>
> On 2013-11-08 13:35, Hafiz Rafibeyli wrote:
>> log on monitor when system hangs was like this:(can send actuall taken screenshot to individual mail adresses)
>>
>> scsi: [ID 107833 kern.warning] WARNING: /pci at 0,0/pci-ide at 1f,2/ide at 0 (ata0):
>> timeout: reset bus, target=0 lun=0
>> scsi: [ID 107833 kern.warning] WARNING: /pci at 0,0/pci-ide at 1f,2/ide at 0 (ata0):
>> timeout: early timeout, target=0 lun=0
>> gda: [ID 107833 kern.warning] WARNING: /pci at 0,0/pci-ide at 1f,2/ide at 0/cmdk at 0,0 (Disk0):
>> Error for command 'read sector' Error Level: Informational
>> gda: [ID 107833 kern.notice] Sense Key: aborted command
>> gda: [ID 107833 kern.notice] Vendor 'Gen-ATA ' error code: 0x3
>> gda: [ID 107833 kern.warning] WARNING: /pci at 0,0/pci-ide at 1f,2/ide at 0/cmdk at 0,0 (Disk0):
>> Error for command 'read sector' Error Level: Informational
>> gda: [ID 107833 kern.notice] Sense Key: aborted command
>> gda: [ID 107833 kern.notice] Vendor 'Gen-ATA ' error code: 0x3
>> scsi: [ID 107833 kern.warning] WARNING: /pci at 0,0/pci-ide at 1f,2/ide at 0 (ata0):
>> timeout: abort request, target=0 lun=0
>> scsi: [ID 107833 kern.warning] WARNING: /pci at 0,0/pci-ide at 1f,2/ide at 0 (ata0):
>> timeout: abort device, target=0 lun=0
>> scsi: [ID 107833 kern.warning] WARNING: /pci at 0,0/pci-ide at 1f,2/ide at 0 (ata0):
>> timeout: reset target, target=0 lun=0
>> scsi: [ID 107833 kern.warning] WARNING: /pci at 0,0/pci-ide at 1f,2/ide at 0 (ata0):
>> timeout: reset bus, target=0 lun=0
>> scsi: [ID 107833 kern.warning] WARNING: /pci at 0,0/pci-ide at 1f,2/ide at 0 (ata0):
>> timeout: early timeout, target=0 lun=0
>> gda: [ID 107833 kern.warning] WARNING: /pci at 0,0/pci-ide at 1f,2/ide at 0/cmdk at 0,0 (Disk0):
>> Error for command 'read sector' Error Level: Informational
>> gda: [ID 107833 kern.notice] Sense Key: aborted command
>> gda: [ID 107833 kern.notice] Vendor 'Gen-ATA ' error code: 0x3
>> gda: [ID 107833 kern.warning] WARNING: /pci at 0,0/pci-ide at 1f,2/ide at 0/cmdk at 0,0 (Disk0):
>>
>>
>> Hello,
>>
>> Omnios version:SunOS 5.11 omnios-b281e50
>> Server:Supermicro X8DAH (24x storage chassis)
>>
>> we are using omnios as a production nfs server for Esxi hosts.
>>
>> everything was ok,but last 20 days system hangs 3 times.Nothing changed on hardware side.
>>
>> for OS disks we are using two SSDSA2SH032G1GN(32 Gb Intel X25-E SSD) in zfs mirror attached onboard sata ports of motherboard.
>>
>> I captured monitor screenshot when system hangs,and sending as attachment.
>>
>>
>> My pools info:
>>
>> pool: rpool
>> state: ONLINE
>> scan: resilvered 20.0G in 0h3m with 0 errors on Sun Oct 20 14:01:01 2013
>> config:
>>
>> NAME STATE READ WRITE CKSUM
>> rpool ONLINE 0 0 0
>> mirror-0 ONLINE 0 0 0
>> c4d0s0 ONLINE 0 0 0
>> c3d1s0 ONLINE 0 0 0
>>
>> errors: No known data errors
>>
>>
>> pool: zpool1
>> state: ONLINE
>> status: Some supported features are not enabled on the pool. The pool can
>> still be used, but some features are unavailable.
>> action: Enable all features using 'zpool upgrade'. Once this is done,
>> the pool may no longer be accessible by software that does not support
>> the features. See zpool-features(5) for details.
>> scan: scrub repaired 0 in 5h0m with 0 errors on Sat Oct 12 19:00:53 2013
>> config:
>>
>> NAME STATE READ WRITE CKSUM
>> zpool1 ONLINE 0 0 0
>> raidz1-0 ONLINE 0 0 0
>> c1t5000C50041E9D9A7d0 ONLINE 0 0 0
>> c1t5000C50041F1A5EFd0 ONLINE 0 0 0
>> c1t5000C5004253FF87d0 ONLINE 0 0 0
>> c1t5000C50055A607E3d0 ONLINE 0 0 0
>> c1t5000C50055A628EFd0 ONLINE 0 0 0
>> c1t5000C50055A62F57d0 ONLINE 0 0 0
>> logs
>> mirror-1 ONLINE 0 0 0
>> c1t5001517959627219d0 ONLINE 0 0 0
>> c1t5001517BB2747BE7d0 ONLINE 0 0 0
>> cache
>> c1t5001517803D007D8d0 ONLINE 0 0 0
>> c1t5001517BB2AFB592d0 ONLINE 0 0 0
>> spares
>> c1t5000C5005600A6B3d0 AVAIL
>> c1t5000C5005600B43Bd0 AVAIL
>>
>> errors: No known data errors
>>
>>
>>
>>
>>
>> _______________________________________________
>> OmniOS-discuss mailing list
>> OmniOS-discuss at lists.omniti.com
>> http://lists.omniti.com/mailman/listinfo/omnios-discuss
>>
>
> _______________________________________________
> OmniOS-discuss mailing list
> OmniOS-discuss at lists.omniti.com
> http://lists.omniti.com/mailman/listinfo/omnios-discuss
--
Richard.Elling at RichardElling.com
+1-760-896-4422
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://omniosce.org/ml-archive/attachments/20131108/7601fda4/attachment.html>
------------------------------
Subject: Digest Footer
_______________________________________________
OmniOS-discuss mailing list
OmniOS-discuss at lists.omniti.com
http://lists.omniti.com/mailman/listinfo/omnios-discuss
------------------------------
End of OmniOS-discuss Digest, Vol 20, Issue 7
*********************************************
More information about the OmniOS-discuss
mailing list