[OmniOS-discuss] zfs/zpool commands have stopped responding

[Eric Sproul]

ZFS is probably unable to make any progress because the hardware is
busy freaking out. :)  Target 41 seems like it is misbehaving and
causing the HBA some indigestion.  If you can identify target 41's
physical location, you could try pulling that device.

On Fri, Jan 18, 2013 at 9:29 AM, Paul Jochum
<paul.jochum at alcatel-lucent.com> wrote:
> Hi All:
>
>     I have an OmniOS server running, under which zfs/zpool commands have
> stopped responding.  Any activities that involve the external JBODs storage
> seem to hang.  Is there a way to kill/reset this other than rebooting the
> server?
>
> Here is some background:
>
> when logging in, the following is displayed:
>
> OmniOS 5.11     omnios-79686dc  2012.03.06
>
> uname -a
>
> SunOS lss-bkup301 5.11 omnios-eae537b i86pc i386 i86pc Solaris
>
> hardware:
>
> server - SUN x4250
> LSI SAS HBAs (I believe there are 5 in the system), they are all of the same
> type (LSI SAS 9200-8e)
> a combination of SUN J4400 JBODs, and one DataOnStorage DNS-1660D
>
> What I am seeing:
>
> The system is responding to input (I can log in, view files in the the root
> pool, perform commands as long as they are restricted to the root pool,
> etc.)
> an "ls" command to filesystems on the external JBODs hang
> zpool status commands on the external JBOD hang
> the following is coming out on the console, rolling about every 10-30
> seconds:
>
> Jan 18 08:20:06 lss-bkup301 scsi: WARNING:
> /pci at 0,0/pci8086,25f9 at 6/pci111d,801c at 0/pci111d,801c at 4/pci1000,3150 at 0 (mpt4):
>
> Jan 18 08:20:06 lss-bkup301     Disconnected command timeout for Target 41
>
> I am seeing a lot of messages in /var/adm/messages, but they all seem to be
> around the following:
>
> Jan 18 08:22:38 lss-bkup301 scsi: [ID 365881 kern.info]
> /pci at 0,0/pci8086,25f9 at 6/pci111d,801c at 0/pci111d,801c at 4/pci1000,3150 at 0 (mpt4):
> Jan 18 08:22:38 lss-bkup301     Log info 0x31130000 received for target 41.
> Jan 18 08:22:38 lss-bkup301     scsi_status=0x0, ioc_status=0x8048,
> scsi_state=0xc
> Jan 18 08:22:38 lss-bkup301 scsi: [ID 243001 kern.warning] WARNING:
> /pci at 0,0/pci8086,25f9 at 6/pci111d,801c at 0/pci111d,801c at 4/pci1000,3150 at 0 (mpt4):
> Jan 18 08:22:38 lss-bkup301     mpt_handle_event_sync: IOCStatus=0x8000,
> IOCLogInfo=0x31111000
> Jan 18 08:22:38 lss-bkup301 scsi: [ID 243001 kern.warning] WARNING:
> /pci at 0,0/pci8086,25f9 at 6/pci111d,801c at 0/pci111d,801c at 4/pci1000,3150 at 0 (mpt4):
> Jan 18 08:22:38 lss-bkup301     mpt_handle_event: IOCStatus=0x8000,
> IOCLogInfo=0x31111000
> Jan 18 08:22:38 lss-bkup301 scsi: [ID 365881 kern.info]
> /pci at 0,0/pci8086,25f8 at 4/pci111d,801c at 0/pci111d,801c at 4/pci1000,3150 at 0 (mpt2):
> Jan 18 08:22:38 lss-bkup301     Log info 0x31111000 received for target 41.
> Jan 18 08:22:38 lss-bkup301     scsi_status=0x0, ioc_status=0x804b,
> scsi_state=0xc
> Jan 18 08:22:38 lss-bkup301 scsi: [ID 243001 kern.warning] WARNING:
> /pci at 0,0/pci8086,25f8 at 4/pci111d,801c at 0/pci111d,801c at 4/pci1000,3150 at 0 (mpt2):
> Jan 18 08:22:38 lss-bkup301     SAS Discovery Error on port 4.
> DiscoveryStatus is DiscoveryStatus is |Unaddressable device found|
> Jan 18 08:22:39 lss-bkup301 scsi: [ID 243001 kern.warning] WARNING:
> /pci at 0,0/pci8086,25f9 at 6/pci111d,801c at 0/pci111d,801c at 4/pci1000,3150 at 0 (mpt4):
> Jan 18 08:22:39 lss-bkup301     SAS Discovery Error on port 4.
> DiscoveryStatus is DiscoveryStatus is |Unaddressable device found|
>
> Thank you for looking at this, and I appreciate any help that can be
> provided.  Please let me know if there is any additional information that
> would help to diagnose this.
>
> Paul
>
>
> _______________________________________________
> OmniOS-discuss mailing list
> OmniOS-discuss at lists.omniti.com
> http://lists.omniti.com/mailman/listinfo/omnios-discuss
>