[OmniOS-discuss] zfs/zpool commands have stopped responding
Paul Jochum
paul.jochum at alcatel-lucent.com
Fri Jan 18 12:44:27 EST 2013
Hi Ryan:
One update, after a few hours of hanging, one of the drives has
been FAULTED, and the system is no longer hanging. But, I would still
be interested in understanding this problem, it is not the first time I
have seen it.
At the end of the cfgadm output, it has:
cfgadm: Configuration administration not supported: Error: hotplug
service is probably not running, please use 'svcadm enable hotplug' to
enable the service. See cfgadm_shp(1M) for more details.
# svcs -a | grep hotplug
disabled Jan_15 svc:/system/hotplug:default
Should this be enabled? Sounds important, but why would it not be
enabled by default?
# cfgadm -la
Ap_Id Type Receptacle Occupant
Condition
c13 scsi-sas connected configured
unknown
c13::0,0 disk-path connected configured
unknown
c13::1,0 disk-path connected configured
unknown
c13::es/ses0 ESI connected configured
unknown
c13::smp/expd0 smp connected configured
unknown
c14 scsi-sas connected configured
unknown
c14::3a,0 disk-path connected configured
unknown
c14::3b,0 disk-path connected configured
unknown
c14::3c,0 disk-path connected configured
unknown
c14::3d,0 disk-path connected configured
unknown
c14::3e,0 disk-path connected configured
unknown
c14::3f,0 disk-path connected configured
unknown
c14::4a,0 disk-path connected configured
unknown
c14::4b,0 disk-path connected configured
unknown
c14::4c,0 disk-path connected configured
unknown
c14::4d,0 disk-path connected configured
unknown
c14::4e,0 disk-path connected configured
unknown
c14::4f,0 disk-path connected configured
unknown
c14::5a,0 disk-path connected configured
unknown
c14::5b,0 disk-path connected configured
unknown
c14::5c,0 disk-path connected configured
unknown
c14::5d,0 disk-path connected configured
unknown
c14::5e,0 disk-path connected configured
unknown
c14::5f,0 disk-path connected configured
unknown
c14::32,0 disk-path connected configured
unknown
c14::33,0 disk-path connected configured
unknown
c14::34,0 disk-path connected configured
unknown
c14::35,0 disk-path connected configured
unknown
c14::36,0 disk-path connected configured
unknown
c14::37,0 disk-path connected configured
unknown
c14::38,0 disk-path connected configured
unknown
c14::39,0 disk-path connected configured
unknown
c14::40,0 disk-path connected configured
unknown
c14::41,0 disk-path connected configured
unknown
c14::42,0 disk-path connected configured
unknown
c14::43,0 disk-path connected configured
unknown
c14::44,0 disk-path connected configured
unknown
c14::45,0 disk-path connected configured
unknown
c14::46,0 disk-path connected configured
unknown
c14::47,0 disk-path connected configured
unknown
c14::48,0 disk-path connected configured
unknown
c14::49,0 disk-path connected configured
unknown
c14::50,0 disk-path connected configured
unknown
c14::51,0 disk-path connected configured
unknown
c14::52,0 disk-path connected configured
unknown
c14::53,0 disk-path connected configured
unknown
c14::54,0 disk-path connected configured
unknown
c14::55,0 disk-path connected configured
unknown
c14::56,0 disk-path connected configured
unknown
c14::57,0 disk-path connected configured
unknown
c14::58,0 disk-path connected configured
unknown
c14::59,0 disk-path connected configured
unknown
c14::60,0 disk-path connected configured
unknown
c14::61,0 disk-path connected configured
unknown
c14::es/ses1 ESI connected configured
unknown
c14::es/ses2 ESI connected configured
unknown
c14::smp/expd1 smp connected configured
unknown
c14::smp/expd2 smp connected configured
unknown
c15 scsi-sas connected configured
unknown
c15::0,0 disk-path connected configured
unknown
c15::1,0 disk-path connected configured
unknown
c15::1a,0 disk-path connected configured
unknown
c15::1b,0 disk-path connected configured
unknown
c15::1c,0 disk-path connected configured
unknown
c15::1d,0 disk-path connected configured
unknown
c15::1e,0 disk-path connected configured
unknown
c15::1f,0 disk-path connected configured
unknown
c15::2,0 disk-path connected configured
unknown
c15::2a,0 disk-path connected configured
unknown
c15::2b,0 disk-path connected configured
unknown
c15::2c,0 disk-path connected configured
unknown
c15::2d,0 disk-path connected configured
unknown
c15::2e,0 disk-path connected configured
unknown
c15::2f,0 disk-path connected configured
unknown
c15::3,0 disk-path connected configured
unknown
c15::4,0 disk-path connected configured
unknown
c15::5,0 disk-path connected configured
unknown
c15::6,0 disk-path connected configured
unknown
c15::7,0 disk-path connected configured
unknown
c15::8,0 disk-path connected configured
unknown
c15::9,0 disk-path connected configured
unknown
c15::10,0 disk-path connected configured
unknown
c15::11,0 disk-path connected configured
unknown
c15::12,0 disk-path connected configured
unknown
c15::13,0 disk-path connected configured
unknown
c15::14,0 disk-path connected configured
unknown
c15::15,0 disk-path connected configured
unknown
c15::16,0 disk-path connected configured
unknown
c15::17,0 disk-path connected configured
unknown
c15::18,0 disk-path connected configured
unknown
c15::19,0 disk-path connected configured
unknown
c15::20,0 disk-path connected configured
unknown
c15::21,0 disk-path connected configured
unknown
c15::22,0 disk-path connected configured
unknown
c15::23,0 disk-path connected configured
unknown
c15::24,0 disk-path connected configured
unknown
c15::25,0 disk-path connected configured
unknown
c15::26,0 disk-path connected configured
unknown
c15::27,0 disk-path connected configured
unknown
c15::28,0 disk-path connected configured
unknown
c15::29,0 disk-path connected configured
unknown
c15::a,0 disk-path connected configured
unknown
c15::b,0 disk-path connected configured
unknown
c15::c,0 disk-path connected configured
unknown
c15::d,0 disk-path connected configured
unknown
c15::e,0 disk-path connected configured
unknown
c15::es/ses3 ESI connected configured
unknown
c15::es/ses4 ESI connected configured
unknown
c15::f,0 disk-path connected configured
unknown
c15::smp/expd3 smp connected configured
unknown
c15::smp/expd4 smp connected configured
unknown
c16 scsi-sas connected configured
unknown
c16::3a,0 disk-path connected configured
unknown
c16::3b,0 disk-path connected configured
unknown
c16::3c,0 disk-path connected configured
unknown
c16::3d,0 disk-path connected configured
unknown
c16::3e,0 disk-path connected configured
unknown
c16::3f,0 disk-path connected configured
unknown
c16::4a,0 disk-path connected configured
unknown
c16::4b,0 disk-path connected configured
unknown
c16::4c,0 disk-path connected configured
unknown
c16::4d,0 disk-path connected configured
unknown
c16::4e,0 disk-path connected configured
unknown
c16::4f,0 disk-path connected configured
unknown
c16::5a,0 disk-path connected configured
unknown
c16::5b,0 disk-path connected configured
unknown
c16::5c,0 disk-path connected configured
unknown
c16::5d,0 disk-path connected configured
unknown
c16::5e,0 disk-path connected configured
unknown
c16::5f,0 disk-path connected configured
unknown
c16::32,0 disk-path connected configured
unknown
c16::33,0 disk-path connected configured
unknown
c16::34,0 disk-path connected configured
unknown
c16::35,0 disk-path connected configured
unknown
c16::36,0 disk-path connected configured
unknown
c16::37,0 disk-path connected configured
unknown
c16::38,0 disk-path connected configured
unknown
c16::39,0 disk-path connected configured
unknown
c16::40,0 disk-path connected configured
unknown
c16::41,0 disk-path connected configured
unknown
c16::42,0 disk-path connected configured
unknown
c16::43,0 disk-path connected configured
unknown
c16::44,0 disk-path connected configured
unknown
c16::45,0 disk-path connected configured
unknown
c16::46,0 disk-path connected configured
unknown
c16::47,0 disk-path connected configured
unknown
c16::48,0 disk-path connected configured
unknown
c16::49,0 disk-path connected configured
unknown
c16::50,0 disk-path connected configured
unknown
c16::51,0 disk-path connected configured
unknown
c16::52,0 disk-path connected configured
unknown
c16::53,0 disk-path connected configured
unknown
c16::54,0 disk-path connected configured
unknown
c16::55,0 disk-path connected configured
unknown
c16::56,0 disk-path connected configured
unknown
c16::57,0 disk-path connected configured
unknown
c16::58,0 disk-path connected configured
unknown
c16::59,0 disk-path connected configured
unknown
c16::60,0 disk-path connected configured
unknown
c16::61,0 disk-path connected configured
unknown
c16::es/ses5 ESI connected configured
unknown
c16::es/ses6 ESI connected configured
unknown
c16::smp/expd5 smp connected configured
unknown
c16::smp/expd6 smp connected configured
unknown
c17 scsi-sas connected configured
unknown
c17::0,0 disk-path connected configured
unknown
c17::1,0 disk-path connected configured
unknown
c17::1a,0 disk-path connected configured
unknown
c17::1b,0 disk-path connected configured
unknown
c17::1c,0 disk-path connected configured
unknown
c17::1d,0 disk-path connected configured
unknown
c17::1e,0 disk-path connected configured
unknown
c17::1f,0 disk-path connected configured
unknown
c17::2,0 disk-path connected configured
unknown
c17::2a,0 disk-path connected configured
unknown
c17::2b,0 disk-path connected configured
unknown
c17::2c,0 disk-path connected configured
unknown
c17::2d,0 disk-path connected configured
unknown
c17::2e,0 disk-path connected configured
unknown
c17::2f,0 disk-path connected configured
unknown
c17::3,0 disk-path connected configured
unknown
c17::4,0 disk-path connected configured
unknown
c17::5,0 disk-path connected configured
unknown
c17::6,0 disk-path connected configured
unknown
c17::7,0 disk-path connected configured
unknown
c17::8,0 disk-path connected configured
unknown
c17::9,0 disk-path connected configured
unknown
c17::10,0 disk-path connected configured
unknown
c17::11,0 disk-path connected configured
unknown
c17::12,0 disk-path connected configured
unknown
c17::13,0 disk-path connected configured
unknown
c17::14,0 disk-path connected configured
unknown
c17::15,0 disk-path connected configured
unknown
c17::16,0 disk-path connected configured
unknown
c17::17,0 disk-path connected configured
unknown
c17::18,0 disk-path connected configured
unknown
c17::19,0 disk-path connected configured
unknown
c17::20,0 disk-path connected configured
unknown
c17::21,0 disk-path connected configured
unknown
c17::22,0 disk-path connected configured
unknown
c17::23,0 disk-path connected configured
unknown
c17::24,0 disk-path connected configured
unknown
c17::25,0 disk-path connected configured
unknown
c17::26,0 disk-path connected configured
unknown
c17::27,0 disk-path connected configured
unknown
c17::28,0 disk-path connected configured
unknown
c17::29,0 disk-path connected configured
unknown
c17::a,0 disk-path connected configured
unknown
c17::b,0 disk-path connected configured
unknown
c17::c,0 disk-path connected configured
unknown
c17::d,0 disk-path connected configured
unknown
c17::e,0 disk-path connected configured
unknown
c17::es/ses7 ESI connected configured
unknown
c17::es/ses8 ESI connected configured
unknown
c17::f,0 disk-path connected configured
unknown
c17::smp/expd7 smp connected configured
unknown
c17::smp/expd8 smp connected configured
unknown
c19 scsi-sas connected configured
unknown
c19::0,0 disk-path connected configured
unknown
c19::1,0 disk-path connected configured
unknown
c19::1a,0 disk-path connected configured
unknown
c19::1b,0 disk-path connected configured
unknown
c19::1c,0 disk-path connected configured
unknown
c19::1d,0 disk-path connected configured
unknown
c19::1e,0 disk-path connected configured
unknown
c19::1f,0 disk-path connected configured
unknown
c19::2,0 disk-path connected configured
unknown
c19::3,0 disk-path connected configured
unknown
c19::4,0 disk-path connected configured
unknown
c19::5,0 disk-path connected configured
unknown
c19::6,0 disk-path connected configured
unknown
c19::7,0 disk-path connected configured
unknown
c19::8,0 disk-path connected configured
unknown
c19::9,0 disk-path connected configured
unknown
c19::10,0 disk-path connected configured
unknown
c19::11,0 disk-path connected configured
unknown
c19::12,0 disk-path connected configured
unknown
c19::13,0 disk-path connected configured
unknown
c19::14,0 disk-path connected configured
unknown
c19::15,0 disk-path connected configured
unknown
c19::16,0 disk-path connected configured
unknown
c19::17,0 disk-path connected configured
unknown
c19::18,0 disk-path connected configured
unknown
c19::19,0 disk-path connected configured
unknown
c19::20,0 disk-path connected configured
unknown
c19::21,0 disk-path connected configured
unknown
c19::22,0 disk-path connected configured
unknown
c19::23,0 disk-path connected configured
unknown
c19::24,0 disk-path connected configured
unknown
c19::25,0 disk-path connected configured
unknown
c19::26,0 disk-path connected configured
unknown
c19::27,0 disk-path connected configured
unknown
c19::a,0 disk-path connected configured
unknown
c19::b,0 disk-path connected configured
unknown
c19::c,0 disk-path connected configured
unknown
c19::d,0 disk-path connected configured
unknown
c19::e,0 disk-path connected configured
unknown
c19::es/ses9 ESI connected configured
unknown
c19::f,0 disk-path connected configured
unknown
c19::smp/expd10 smp connected configured
unknown
c19::smp/expd11 smp connected configured
unknown
c19::smp/expd12 smp connected configured
unknown
sata3/0 sata-port empty unconfigured ok
sata3/1 sata-port empty unconfigured ok
sata3/2 sata-port empty unconfigured ok
sata3/3 sata-port empty unconfigured ok
sata3/4 sata-port empty unconfigured ok
sata3/5 sata-port empty unconfigured ok
usb8/1 unknown empty unconfigured ok
usb8/2 unknown empty unconfigured ok
usb9/1 unknown empty unconfigured ok
usb9/2 usb-device connected configured ok
usb10/1 unknown empty unconfigured ok
usb10/2 unknown empty unconfigured ok
usb11/1 unknown empty unconfigured ok
usb11/2 unknown empty unconfigured ok
usb12/1 unknown empty unconfigured ok
usb12/2 unknown empty unconfigured ok
usb12/3 unknown empty unconfigured ok
usb12/4 unknown empty unconfigured ok
usb12/5 unknown empty unconfigured ok
usb12/6 usb-hub connected configured ok
usb12/6.1 unknown empty unconfigured ok
usb12/6.2 unknown empty unconfigured ok
usb12/6.3 unknown empty unconfigured ok
usb12/6.4 unknown empty unconfigured ok
usb12/7 unknown empty unconfigured ok
usb12/8 unknown empty unconfigured ok
cfgadm: Configuration administration not supported: Error: hotplug
service is probably not running, please use 'svcadm enable hotplug' to
enable the service. See cfgadm_shp(1M) for more details.
#
thanks,
Paul
On 01/18/2013 10:27 AM, Ryan Kohler wrote:
> What is the output of:
>
> cfgadm -la
>
> On 1/18/2013 7:46 AM, Paul Jochum wrote:
>> Thanks Eric. Any suggestions on how to convert Target 41 into either
>> an sd
>> device or cXtXdX type name? I assume that this is a single device
>> (such as a
>> single disk), but maybe my assumption is incorrect. I have tried the
>> following in the /dev/rdsk directory:
>>
>> ls -al | grep
>> "/pci at 0,0/pci8086,25f8 at 4/pci111d,801c at 0/pci111d,801c at 4/pci1000,3150 at 0"
>>
>> but this returns 1008 devices, it seems to narrow it down to c15tXd0,
>> where X
>> is from 0 to 47.
>>
>> thanks,
>>
>> Paul
>>
>> On 01/18/2013 09:03 AM, Eric Sproul wrote:
>>> ZFS is probably unable to make any progress because the hardware is
>>> busy freaking out. :) Target 41 seems like it is misbehaving and
>>> causing the HBA some indigestion. If you can identify target 41's
>>> physical location, you could try pulling that device.
>>>
>>> On Fri, Jan 18, 2013 at 9:29 AM, Paul Jochum
>>> <paul.jochum at alcatel-lucent.com> wrote:
>>>> Hi All:
>>>>
>>>> I have an OmniOS server running, under which zfs/zpool
>>>> commands have
>>>> stopped responding. Any activities that involve the external JBODs
>>>> storage
>>>> seem to hang. Is there a way to kill/reset this other than
>>>> rebooting the
>>>> server?
>>>>
>>>> Here is some background:
>>>>
>>>> when logging in, the following is displayed:
>>>>
>>>> OmniOS 5.11 omnios-79686dc 2012.03.06
>>>>
>>>> uname -a
>>>>
>>>> SunOS lss-bkup301 5.11 omnios-eae537b i86pc i386 i86pc Solaris
>>>>
>>>> hardware:
>>>>
>>>> server - SUN x4250
>>>> LSI SAS HBAs (I believe there are 5 in the system), they are all of
>>>> the same
>>>> type (LSI SAS 9200-8e)
>>>> a combination of SUN J4400 JBODs, and one DataOnStorage DNS-1660D
>>>>
>>>> What I am seeing:
>>>>
>>>> The system is responding to input (I can log in, view files in the
>>>> the root
>>>> pool, perform commands as long as they are restricted to the root
>>>> pool,
>>>> etc.)
>>>> an "ls" command to filesystems on the external JBODs hang
>>>> zpool status commands on the external JBOD hang
>>>> the following is coming out on the console, rolling about every 10-30
>>>> seconds:
>>>>
>>>> Jan 18 08:20:06 lss-bkup301 scsi: WARNING:
>>>> /pci at 0,0/pci8086,25f9 at 6/pci111d,801c at 0/pci111d,801c at 4/pci1000,3150 at 0 (mpt4):
>>>>
>>>>
>>>> Jan 18 08:20:06 lss-bkup301 Disconnected command timeout for
>>>> Target 41
>>>>
>>>> I am seeing a lot of messages in /var/adm/messages, but they all
>>>> seem to be
>>>> around the following:
>>>>
>>>> Jan 18 08:22:38 lss-bkup301 scsi: [ID 365881 kern.info]
>>>> /pci at 0,0/pci8086,25f9 at 6/pci111d,801c at 0/pci111d,801c at 4/pci1000,3150 at 0 (mpt4):
>>>>
>>>> Jan 18 08:22:38 lss-bkup301 Log info 0x31130000 received for
>>>> target 41.
>>>> Jan 18 08:22:38 lss-bkup301 scsi_status=0x0, ioc_status=0x8048,
>>>> scsi_state=0xc
>>>> Jan 18 08:22:38 lss-bkup301 scsi: [ID 243001 kern.warning] WARNING:
>>>> /pci at 0,0/pci8086,25f9 at 6/pci111d,801c at 0/pci111d,801c at 4/pci1000,3150 at 0 (mpt4):
>>>>
>>>> Jan 18 08:22:38 lss-bkup301 mpt_handle_event_sync:
>>>> IOCStatus=0x8000,
>>>> IOCLogInfo=0x31111000
>>>> Jan 18 08:22:38 lss-bkup301 scsi: [ID 243001 kern.warning] WARNING:
>>>> /pci at 0,0/pci8086,25f9 at 6/pci111d,801c at 0/pci111d,801c at 4/pci1000,3150 at 0 (mpt4):
>>>>
>>>> Jan 18 08:22:38 lss-bkup301 mpt_handle_event: IOCStatus=0x8000,
>>>> IOCLogInfo=0x31111000
>>>> Jan 18 08:22:38 lss-bkup301 scsi: [ID 365881 kern.info]
>>>> /pci at 0,0/pci8086,25f8 at 4/pci111d,801c at 0/pci111d,801c at 4/pci1000,3150 at 0 (mpt2):
>>>>
>>>> Jan 18 08:22:38 lss-bkup301 Log info 0x31111000 received for
>>>> target 41.
>>>> Jan 18 08:22:38 lss-bkup301 scsi_status=0x0, ioc_status=0x804b,
>>>> scsi_state=0xc
>>>> Jan 18 08:22:38 lss-bkup301 scsi: [ID 243001 kern.warning] WARNING:
>>>> /pci at 0,0/pci8086,25f8 at 4/pci111d,801c at 0/pci111d,801c at 4/pci1000,3150 at 0 (mpt2):
>>>>
>>>> Jan 18 08:22:38 lss-bkup301 SAS Discovery Error on port 4.
>>>> DiscoveryStatus is DiscoveryStatus is |Unaddressable device found|
>>>> Jan 18 08:22:39 lss-bkup301 scsi: [ID 243001 kern.warning] WARNING:
>>>> /pci at 0,0/pci8086,25f9 at 6/pci111d,801c at 0/pci111d,801c at 4/pci1000,3150 at 0 (mpt4):
>>>>
>>>> Jan 18 08:22:39 lss-bkup301 SAS Discovery Error on port 4.
>>>> DiscoveryStatus is DiscoveryStatus is |Unaddressable device found|
>>>>
>>>> Thank you for looking at this, and I appreciate any help that can be
>>>> provided. Please let me know if there is any additional
>>>> information that
>>>> would help to diagnose this.
>>>>
>>>> Paul
>>>>
>>>>
>>>> _______________________________________________
>>>> OmniOS-discuss mailing list
>>>> OmniOS-discuss at lists.omniti.com
>>>> http://lists.omniti.com/mailman/listinfo/omnios-discuss
>>>>
>> _______________________________________________
>> OmniOS-discuss mailing list
>> OmniOS-discuss at lists.omniti.com
>> http://lists.omniti.com/mailman/listinfo/omnios-discuss
> _______________________________________________
> OmniOS-discuss mailing list
> OmniOS-discuss at lists.omniti.com
> http://lists.omniti.com/mailman/listinfo/omnios-discuss
More information about the OmniOS-discuss
mailing list