[OmniOS-discuss] zfs/zpool commands have stopped responding
Ryan Kohler
kohlerr at ics.uci.edu
Fri Jan 18 12:57:42 EST 2013
Looks like I meant:
cfgadm -lav
Sorry about that! That should hopefully help you find which drive is faulted
when looking at the output and comparing it with the errors messages you have.
My hotplug is disabled so I'm not sure if it's necessary:
svcs hotplug
STATE STIME FMRI
disabled Sep_27 svc:/system/hotplug:default
Ryan
On 1/18/2013 9:44 AM, Paul Jochum wrote:
> Hi Ryan:
>
> One update, after a few hours of hanging, one of the drives has been
> FAULTED, and the system is no longer hanging. But, I would still be
> interested in understanding this problem, it is not the first time I have seen
> it.
>
> At the end of the cfgadm output, it has:
>
> cfgadm: Configuration administration not supported: Error: hotplug service is
> probably not running, please use 'svcadm enable hotplug' to enable the
> service. See cfgadm_shp(1M) for more details.
> # svcs -a | grep hotplug
> disabled Jan_15 svc:/system/hotplug:default
>
> Should this be enabled? Sounds important, but why would it not be enabled by
> default?
>
>
> # cfgadm -la
> Ap_Id Type Receptacle Occupant Condition
> c13 scsi-sas connected configured unknown
> c13::0,0 disk-path connected configured unknown
> c13::1,0 disk-path connected configured unknown
> c13::es/ses0 ESI connected configured unknown
> c13::smp/expd0 smp connected configured unknown
> c14 scsi-sas connected configured unknown
> c14::3a,0 disk-path connected configured unknown
> c14::3b,0 disk-path connected configured unknown
> c14::3c,0 disk-path connected configured unknown
> c14::3d,0 disk-path connected configured unknown
> c14::3e,0 disk-path connected configured unknown
> c14::3f,0 disk-path connected configured unknown
> c14::4a,0 disk-path connected configured unknown
> c14::4b,0 disk-path connected configured unknown
> c14::4c,0 disk-path connected configured unknown
> c14::4d,0 disk-path connected configured unknown
> c14::4e,0 disk-path connected configured unknown
> c14::4f,0 disk-path connected configured unknown
> c14::5a,0 disk-path connected configured unknown
> c14::5b,0 disk-path connected configured unknown
> c14::5c,0 disk-path connected configured unknown
> c14::5d,0 disk-path connected configured unknown
> c14::5e,0 disk-path connected configured unknown
> c14::5f,0 disk-path connected configured unknown
> c14::32,0 disk-path connected configured unknown
> c14::33,0 disk-path connected configured unknown
> c14::34,0 disk-path connected configured unknown
> c14::35,0 disk-path connected configured unknown
> c14::36,0 disk-path connected configured unknown
> c14::37,0 disk-path connected configured unknown
> c14::38,0 disk-path connected configured unknown
> c14::39,0 disk-path connected configured unknown
> c14::40,0 disk-path connected configured unknown
> c14::41,0 disk-path connected configured unknown
> c14::42,0 disk-path connected configured unknown
> c14::43,0 disk-path connected configured unknown
> c14::44,0 disk-path connected configured unknown
> c14::45,0 disk-path connected configured unknown
> c14::46,0 disk-path connected configured unknown
> c14::47,0 disk-path connected configured unknown
> c14::48,0 disk-path connected configured unknown
> c14::49,0 disk-path connected configured unknown
> c14::50,0 disk-path connected configured unknown
> c14::51,0 disk-path connected configured unknown
> c14::52,0 disk-path connected configured unknown
> c14::53,0 disk-path connected configured unknown
> c14::54,0 disk-path connected configured unknown
> c14::55,0 disk-path connected configured unknown
> c14::56,0 disk-path connected configured unknown
> c14::57,0 disk-path connected configured unknown
> c14::58,0 disk-path connected configured unknown
> c14::59,0 disk-path connected configured unknown
> c14::60,0 disk-path connected configured unknown
> c14::61,0 disk-path connected configured unknown
> c14::es/ses1 ESI connected configured unknown
> c14::es/ses2 ESI connected configured unknown
> c14::smp/expd1 smp connected configured unknown
> c14::smp/expd2 smp connected configured unknown
> c15 scsi-sas connected configured unknown
> c15::0,0 disk-path connected configured unknown
> c15::1,0 disk-path connected configured unknown
> c15::1a,0 disk-path connected configured unknown
> c15::1b,0 disk-path connected configured unknown
> c15::1c,0 disk-path connected configured unknown
> c15::1d,0 disk-path connected configured unknown
> c15::1e,0 disk-path connected configured unknown
> c15::1f,0 disk-path connected configured unknown
> c15::2,0 disk-path connected configured unknown
> c15::2a,0 disk-path connected configured unknown
> c15::2b,0 disk-path connected configured unknown
> c15::2c,0 disk-path connected configured unknown
> c15::2d,0 disk-path connected configured unknown
> c15::2e,0 disk-path connected configured unknown
> c15::2f,0 disk-path connected configured unknown
> c15::3,0 disk-path connected configured unknown
> c15::4,0 disk-path connected configured unknown
> c15::5,0 disk-path connected configured unknown
> c15::6,0 disk-path connected configured unknown
> c15::7,0 disk-path connected configured unknown
> c15::8,0 disk-path connected configured unknown
> c15::9,0 disk-path connected configured unknown
> c15::10,0 disk-path connected configured unknown
> c15::11,0 disk-path connected configured unknown
> c15::12,0 disk-path connected configured unknown
> c15::13,0 disk-path connected configured unknown
> c15::14,0 disk-path connected configured unknown
> c15::15,0 disk-path connected configured unknown
> c15::16,0 disk-path connected configured unknown
> c15::17,0 disk-path connected configured unknown
> c15::18,0 disk-path connected configured unknown
> c15::19,0 disk-path connected configured unknown
> c15::20,0 disk-path connected configured unknown
> c15::21,0 disk-path connected configured unknown
> c15::22,0 disk-path connected configured unknown
> c15::23,0 disk-path connected configured unknown
> c15::24,0 disk-path connected configured unknown
> c15::25,0 disk-path connected configured unknown
> c15::26,0 disk-path connected configured unknown
> c15::27,0 disk-path connected configured unknown
> c15::28,0 disk-path connected configured unknown
> c15::29,0 disk-path connected configured unknown
> c15::a,0 disk-path connected configured unknown
> c15::b,0 disk-path connected configured unknown
> c15::c,0 disk-path connected configured unknown
> c15::d,0 disk-path connected configured unknown
> c15::e,0 disk-path connected configured unknown
> c15::es/ses3 ESI connected configured unknown
> c15::es/ses4 ESI connected configured unknown
> c15::f,0 disk-path connected configured unknown
> c15::smp/expd3 smp connected configured unknown
> c15::smp/expd4 smp connected configured unknown
> c16 scsi-sas connected configured unknown
> c16::3a,0 disk-path connected configured unknown
> c16::3b,0 disk-path connected configured unknown
> c16::3c,0 disk-path connected configured unknown
> c16::3d,0 disk-path connected configured unknown
> c16::3e,0 disk-path connected configured unknown
> c16::3f,0 disk-path connected configured unknown
> c16::4a,0 disk-path connected configured unknown
> c16::4b,0 disk-path connected configured unknown
> c16::4c,0 disk-path connected configured unknown
> c16::4d,0 disk-path connected configured unknown
> c16::4e,0 disk-path connected configured unknown
> c16::4f,0 disk-path connected configured unknown
> c16::5a,0 disk-path connected configured unknown
> c16::5b,0 disk-path connected configured unknown
> c16::5c,0 disk-path connected configured unknown
> c16::5d,0 disk-path connected configured unknown
> c16::5e,0 disk-path connected configured unknown
> c16::5f,0 disk-path connected configured unknown
> c16::32,0 disk-path connected configured unknown
> c16::33,0 disk-path connected configured unknown
> c16::34,0 disk-path connected configured unknown
> c16::35,0 disk-path connected configured unknown
> c16::36,0 disk-path connected configured unknown
> c16::37,0 disk-path connected configured unknown
> c16::38,0 disk-path connected configured unknown
> c16::39,0 disk-path connected configured unknown
> c16::40,0 disk-path connected configured unknown
> c16::41,0 disk-path connected configured unknown
> c16::42,0 disk-path connected configured unknown
> c16::43,0 disk-path connected configured unknown
> c16::44,0 disk-path connected configured unknown
> c16::45,0 disk-path connected configured unknown
> c16::46,0 disk-path connected configured unknown
> c16::47,0 disk-path connected configured unknown
> c16::48,0 disk-path connected configured unknown
> c16::49,0 disk-path connected configured unknown
> c16::50,0 disk-path connected configured unknown
> c16::51,0 disk-path connected configured unknown
> c16::52,0 disk-path connected configured unknown
> c16::53,0 disk-path connected configured unknown
> c16::54,0 disk-path connected configured unknown
> c16::55,0 disk-path connected configured unknown
> c16::56,0 disk-path connected configured unknown
> c16::57,0 disk-path connected configured unknown
> c16::58,0 disk-path connected configured unknown
> c16::59,0 disk-path connected configured unknown
> c16::60,0 disk-path connected configured unknown
> c16::61,0 disk-path connected configured unknown
> c16::es/ses5 ESI connected configured unknown
> c16::es/ses6 ESI connected configured unknown
> c16::smp/expd5 smp connected configured unknown
> c16::smp/expd6 smp connected configured unknown
> c17 scsi-sas connected configured unknown
> c17::0,0 disk-path connected configured unknown
> c17::1,0 disk-path connected configured unknown
> c17::1a,0 disk-path connected configured unknown
> c17::1b,0 disk-path connected configured unknown
> c17::1c,0 disk-path connected configured unknown
> c17::1d,0 disk-path connected configured unknown
> c17::1e,0 disk-path connected configured unknown
> c17::1f,0 disk-path connected configured unknown
> c17::2,0 disk-path connected configured unknown
> c17::2a,0 disk-path connected configured unknown
> c17::2b,0 disk-path connected configured unknown
> c17::2c,0 disk-path connected configured unknown
> c17::2d,0 disk-path connected configured unknown
> c17::2e,0 disk-path connected configured unknown
> c17::2f,0 disk-path connected configured unknown
> c17::3,0 disk-path connected configured unknown
> c17::4,0 disk-path connected configured unknown
> c17::5,0 disk-path connected configured unknown
> c17::6,0 disk-path connected configured unknown
> c17::7,0 disk-path connected configured unknown
> c17::8,0 disk-path connected configured unknown
> c17::9,0 disk-path connected configured unknown
> c17::10,0 disk-path connected configured unknown
> c17::11,0 disk-path connected configured unknown
> c17::12,0 disk-path connected configured unknown
> c17::13,0 disk-path connected configured unknown
> c17::14,0 disk-path connected configured unknown
> c17::15,0 disk-path connected configured unknown
> c17::16,0 disk-path connected configured unknown
> c17::17,0 disk-path connected configured unknown
> c17::18,0 disk-path connected configured unknown
> c17::19,0 disk-path connected configured unknown
> c17::20,0 disk-path connected configured unknown
> c17::21,0 disk-path connected configured unknown
> c17::22,0 disk-path connected configured unknown
> c17::23,0 disk-path connected configured unknown
> c17::24,0 disk-path connected configured unknown
> c17::25,0 disk-path connected configured unknown
> c17::26,0 disk-path connected configured unknown
> c17::27,0 disk-path connected configured unknown
> c17::28,0 disk-path connected configured unknown
> c17::29,0 disk-path connected configured unknown
> c17::a,0 disk-path connected configured unknown
> c17::b,0 disk-path connected configured unknown
> c17::c,0 disk-path connected configured unknown
> c17::d,0 disk-path connected configured unknown
> c17::e,0 disk-path connected configured unknown
> c17::es/ses7 ESI connected configured unknown
> c17::es/ses8 ESI connected configured unknown
> c17::f,0 disk-path connected configured unknown
> c17::smp/expd7 smp connected configured unknown
> c17::smp/expd8 smp connected configured unknown
> c19 scsi-sas connected configured unknown
> c19::0,0 disk-path connected configured unknown
> c19::1,0 disk-path connected configured unknown
> c19::1a,0 disk-path connected configured unknown
> c19::1b,0 disk-path connected configured unknown
> c19::1c,0 disk-path connected configured unknown
> c19::1d,0 disk-path connected configured unknown
> c19::1e,0 disk-path connected configured unknown
> c19::1f,0 disk-path connected configured unknown
> c19::2,0 disk-path connected configured unknown
> c19::3,0 disk-path connected configured unknown
> c19::4,0 disk-path connected configured unknown
> c19::5,0 disk-path connected configured unknown
> c19::6,0 disk-path connected configured unknown
> c19::7,0 disk-path connected configured unknown
> c19::8,0 disk-path connected configured unknown
> c19::9,0 disk-path connected configured unknown
> c19::10,0 disk-path connected configured unknown
> c19::11,0 disk-path connected configured unknown
> c19::12,0 disk-path connected configured unknown
> c19::13,0 disk-path connected configured unknown
> c19::14,0 disk-path connected configured unknown
> c19::15,0 disk-path connected configured unknown
> c19::16,0 disk-path connected configured unknown
> c19::17,0 disk-path connected configured unknown
> c19::18,0 disk-path connected configured unknown
> c19::19,0 disk-path connected configured unknown
> c19::20,0 disk-path connected configured unknown
> c19::21,0 disk-path connected configured unknown
> c19::22,0 disk-path connected configured unknown
> c19::23,0 disk-path connected configured unknown
> c19::24,0 disk-path connected configured unknown
> c19::25,0 disk-path connected configured unknown
> c19::26,0 disk-path connected configured unknown
> c19::27,0 disk-path connected configured unknown
> c19::a,0 disk-path connected configured unknown
> c19::b,0 disk-path connected configured unknown
> c19::c,0 disk-path connected configured unknown
> c19::d,0 disk-path connected configured unknown
> c19::e,0 disk-path connected configured unknown
> c19::es/ses9 ESI connected configured unknown
> c19::f,0 disk-path connected configured unknown
> c19::smp/expd10 smp connected configured unknown
> c19::smp/expd11 smp connected configured unknown
> c19::smp/expd12 smp connected configured unknown
> sata3/0 sata-port empty unconfigured ok
> sata3/1 sata-port empty unconfigured ok
> sata3/2 sata-port empty unconfigured ok
> sata3/3 sata-port empty unconfigured ok
> sata3/4 sata-port empty unconfigured ok
> sata3/5 sata-port empty unconfigured ok
> usb8/1 unknown empty unconfigured ok
> usb8/2 unknown empty unconfigured ok
> usb9/1 unknown empty unconfigured ok
> usb9/2 usb-device connected configured ok
> usb10/1 unknown empty unconfigured ok
> usb10/2 unknown empty unconfigured ok
> usb11/1 unknown empty unconfigured ok
> usb11/2 unknown empty unconfigured ok
> usb12/1 unknown empty unconfigured ok
> usb12/2 unknown empty unconfigured ok
> usb12/3 unknown empty unconfigured ok
> usb12/4 unknown empty unconfigured ok
> usb12/5 unknown empty unconfigured ok
> usb12/6 usb-hub connected configured ok
> usb12/6.1 unknown empty unconfigured ok
> usb12/6.2 unknown empty unconfigured ok
> usb12/6.3 unknown empty unconfigured ok
> usb12/6.4 unknown empty unconfigured ok
> usb12/7 unknown empty unconfigured ok
> usb12/8 unknown empty unconfigured ok
> cfgadm: Configuration administration not supported: Error: hotplug service is
> probably not running, please use 'svcadm enable hotplug' to enable the
> service. See cfgadm_shp(1M) for more details.
> #
>
> thanks,
>
> Paul
>
> On 01/18/2013 10:27 AM, Ryan Kohler wrote:
>> What is the output of:
>>
>> cfgadm -la
>>
>> On 1/18/2013 7:46 AM, Paul Jochum wrote:
>>> Thanks Eric. Any suggestions on how to convert Target 41 into either an sd
>>> device or cXtXdX type name? I assume that this is a single device (such as a
>>> single disk), but maybe my assumption is incorrect. I have tried the
>>> following in the /dev/rdsk directory:
>>>
>>> ls -al | grep
>>> "/pci at 0,0/pci8086,25f8 at 4/pci111d,801c at 0/pci111d,801c at 4/pci1000,3150 at 0"
>>>
>>> but this returns 1008 devices, it seems to narrow it down to c15tXd0, where X
>>> is from 0 to 47.
>>>
>>> thanks,
>>>
>>> Paul
>>>
>>> On 01/18/2013 09:03 AM, Eric Sproul wrote:
>>>> ZFS is probably unable to make any progress because the hardware is
>>>> busy freaking out. :) Target 41 seems like it is misbehaving and
>>>> causing the HBA some indigestion. If you can identify target 41's
>>>> physical location, you could try pulling that device.
>>>>
>>>> On Fri, Jan 18, 2013 at 9:29 AM, Paul Jochum
>>>> <paul.jochum at alcatel-lucent.com> wrote:
>>>>> Hi All:
>>>>>
>>>>> I have an OmniOS server running, under which zfs/zpool commands have
>>>>> stopped responding. Any activities that involve the external JBODs storage
>>>>> seem to hang. Is there a way to kill/reset this other than rebooting the
>>>>> server?
>>>>>
>>>>> Here is some background:
>>>>>
>>>>> when logging in, the following is displayed:
>>>>>
>>>>> OmniOS 5.11 omnios-79686dc 2012.03.06
>>>>>
>>>>> uname -a
>>>>>
>>>>> SunOS lss-bkup301 5.11 omnios-eae537b i86pc i386 i86pc Solaris
>>>>>
>>>>> hardware:
>>>>>
>>>>> server - SUN x4250
>>>>> LSI SAS HBAs (I believe there are 5 in the system), they are all of the same
>>>>> type (LSI SAS 9200-8e)
>>>>> a combination of SUN J4400 JBODs, and one DataOnStorage DNS-1660D
>>>>>
>>>>> What I am seeing:
>>>>>
>>>>> The system is responding to input (I can log in, view files in the the root
>>>>> pool, perform commands as long as they are restricted to the root pool,
>>>>> etc.)
>>>>> an "ls" command to filesystems on the external JBODs hang
>>>>> zpool status commands on the external JBOD hang
>>>>> the following is coming out on the console, rolling about every 10-30
>>>>> seconds:
>>>>>
>>>>> Jan 18 08:20:06 lss-bkup301 scsi: WARNING:
>>>>> /pci at 0,0/pci8086,25f9 at 6/pci111d,801c at 0/pci111d,801c at 4/pci1000,3150 at 0 (mpt4):
>>>>>
>>>>> Jan 18 08:20:06 lss-bkup301 Disconnected command timeout for Target 41
>>>>>
>>>>> I am seeing a lot of messages in /var/adm/messages, but they all seem to be
>>>>> around the following:
>>>>>
>>>>> Jan 18 08:22:38 lss-bkup301 scsi: [ID 365881 kern.info]
>>>>> /pci at 0,0/pci8086,25f9 at 6/pci111d,801c at 0/pci111d,801c at 4/pci1000,3150 at 0 (mpt4):
>>>>> Jan 18 08:22:38 lss-bkup301 Log info 0x31130000 received for target 41.
>>>>> Jan 18 08:22:38 lss-bkup301 scsi_status=0x0, ioc_status=0x8048,
>>>>> scsi_state=0xc
>>>>> Jan 18 08:22:38 lss-bkup301 scsi: [ID 243001 kern.warning] WARNING:
>>>>> /pci at 0,0/pci8086,25f9 at 6/pci111d,801c at 0/pci111d,801c at 4/pci1000,3150 at 0 (mpt4):
>>>>> Jan 18 08:22:38 lss-bkup301 mpt_handle_event_sync: IOCStatus=0x8000,
>>>>> IOCLogInfo=0x31111000
>>>>> Jan 18 08:22:38 lss-bkup301 scsi: [ID 243001 kern.warning] WARNING:
>>>>> /pci at 0,0/pci8086,25f9 at 6/pci111d,801c at 0/pci111d,801c at 4/pci1000,3150 at 0 (mpt4):
>>>>> Jan 18 08:22:38 lss-bkup301 mpt_handle_event: IOCStatus=0x8000,
>>>>> IOCLogInfo=0x31111000
>>>>> Jan 18 08:22:38 lss-bkup301 scsi: [ID 365881 kern.info]
>>>>> /pci at 0,0/pci8086,25f8 at 4/pci111d,801c at 0/pci111d,801c at 4/pci1000,3150 at 0 (mpt2):
>>>>> Jan 18 08:22:38 lss-bkup301 Log info 0x31111000 received for target 41.
>>>>> Jan 18 08:22:38 lss-bkup301 scsi_status=0x0, ioc_status=0x804b,
>>>>> scsi_state=0xc
>>>>> Jan 18 08:22:38 lss-bkup301 scsi: [ID 243001 kern.warning] WARNING:
>>>>> /pci at 0,0/pci8086,25f8 at 4/pci111d,801c at 0/pci111d,801c at 4/pci1000,3150 at 0 (mpt2):
>>>>> Jan 18 08:22:38 lss-bkup301 SAS Discovery Error on port 4.
>>>>> DiscoveryStatus is DiscoveryStatus is |Unaddressable device found|
>>>>> Jan 18 08:22:39 lss-bkup301 scsi: [ID 243001 kern.warning] WARNING:
>>>>> /pci at 0,0/pci8086,25f9 at 6/pci111d,801c at 0/pci111d,801c at 4/pci1000,3150 at 0 (mpt4):
>>>>> Jan 18 08:22:39 lss-bkup301 SAS Discovery Error on port 4.
>>>>> DiscoveryStatus is DiscoveryStatus is |Unaddressable device found|
>>>>>
>>>>> Thank you for looking at this, and I appreciate any help that can be
>>>>> provided. Please let me know if there is any additional information that
>>>>> would help to diagnose this.
>>>>>
>>>>> Paul
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> OmniOS-discuss mailing list
>>>>> OmniOS-discuss at lists.omniti.com
>>>>> http://lists.omniti.com/mailman/listinfo/omnios-discuss
>>>>>
>>> _______________________________________________
>>> OmniOS-discuss mailing list
>>> OmniOS-discuss at lists.omniti.com
>>> http://lists.omniti.com/mailman/listinfo/omnios-discuss
>> _______________________________________________
>> OmniOS-discuss mailing list
>> OmniOS-discuss at lists.omniti.com
>> http://lists.omniti.com/mailman/listinfo/omnios-discuss
> _______________________________________________
> OmniOS-discuss mailing list
> OmniOS-discuss at lists.omniti.com
> http://lists.omniti.com/mailman/listinfo/omnios-discuss
More information about the OmniOS-discuss
mailing list