[OmniOS-discuss] [discuss] COMSTAR hanging

[Brian Hechinger]

In the meantime I’ve removed the SLOG and L2ARC just in case. I don’t think that’s it though. At least will have some sort of data point to work with here. :)

-brian

> On Jan 12, 2016, at 10:55 PM, Brian Hechinger <wonko at 4amlunch.net> wrote:
> 
> Ok, it has happened.
> 
> Checking this here, the pool seems to be fine. I can read and write files.
> 
> except ‘zpool status’ is now currently hanging. I can still read/write from the pool, however.
> 
> I can telnet to port 3260, but restarting target services has hung.
> 
> root at basket1:/tank/Share# svcs -a | grep stmf
> online         Jan_05   svc:/system/stmf:default
> root at basket1:/tank/Share# svcs -a | grep target
> disabled       Jan_05   svc:/system/fcoe_target:default
> online         Jan_05   svc:/network/iscsi/target:default
> online         Jan_05   svc:/system/ibsrp/target:default
> root at basket1:/tank/Share# svcadm restart /system/ibsrp/target
> root at basket1:/tank/Share# svcadm restart /network/iscsi/target
> root at basket1:/tank/Share# svcadm restart /system/stmf
> root at basket1:/tank/Share# svcs -a | grep target
> disabled       Jan_05   svc:/system/fcoe_target:default
> online*        22:43:03 svc:/system/ibsrp/target:default
> online*        22:43:13 svc:/network/iscsi/target:default
> root at basket1:/tank/Share# svcs -a | grep stmf
> online*        22:43:18 svc:/system/stmf:default
> root at basket1:/tank/Share#
> 
> I’m doing a crash dump reboot. I’ll post the output somewhere.
> 
> The output of echo '$<threadlist' | mdb -k is attached.
> 
> <threadlist.out>
> 
>> On Jan 8, 2016, at 3:11 PM, Matej Zerovnik <matej at zunaj.si> wrote:
>> 
>> Is the pool usable during comstar hang?
>> Can you write and read from the pool (test both, in my case, when pool froze, I wasn’t able to write to the pool, but I could read).
>> 
>> Again, this might not be connected with Comstar, but in my case, Comstar and pool hang were exchanging.
>> 
>> Matej
>> 
>>> On 08 Jan 2016, at 20:11, Brian Hechinger <wonko at 4amlunch.net> wrote:
>>> 
>>> Yeah, I’m using the 1068E to boot from (this has been supported since before Illumos) but that doesn’t have anything accessed by COMSTAR.
>>> 
>>> It’s the ICH10R SATA that hosts the disks that COMSTAR shares out space from.
>>> 
>>> -brian
>>> 
>>>> On Jan 8, 2016, at 1:31 PM, Richard Jahnel <rjahnel at ellipseinc.com> wrote:
>>>> 
>>>> First off, love SuperMicro good choice IMHO.
>>>> 
>>>> This board has two on board controllers.
>>>> 
>>>> LSI SAS1068E (not 100% sure there are working illumos drivers for this one)
>>>> 
>>>> And
>>>> 
>>>> Intel ICH10R SATA (So I'm guessing your using this one.)
>>>> 
>>>> -----Original Message-----
>>>> From: OmniOS-discuss [mailto:omnios-discuss-bounces at lists.omniti.com] On Behalf Of Brian Hechinger
>>>> Sent: Friday, January 08, 2016 12:16 PM
>>>> To: Matej Zerovnik <matej at zunaj.si>
>>>> Cc: omnios-discuss <omnios-discuss at lists.omniti.com>
>>>> Subject: Re: [OmniOS-discuss] [discuss] COMSTAR hanging
>>>> 
>>>> 
>>>>> Which controller exactly do you have?
>>>> 
>>>> Whatever ACHI stuff is built into the motherboard. Motherboard is X8DTL-3F.
>>>> 
>>>>> Do you know firmware version?
>>>> 
>>>> I’m assuming this is linked to the BIOS version?
>>>> 
>>>>> Which hard drives?
>>>> 
>>>> Hitachi-HUA723030ALA640-MKAOAA50-2.73TB
>>>> 
>>>>> It might not tell much, but it’s good to have as much information as possible.
>>>>> 
>>>>> When comstar hangs, can you telnet to the iSCSI port?
>>>>> What does svcs says, is the service running?
>>>>> What happens in you try to restart it?
>>>>> How do you restart it?
>>>> 
>>>> I’ll try all these things next time.
>>>> 
>>>>> In my case, svcs reported service running, but when I tried to telnet, there was no connection as well as there was no listening port opened when checking with 'netstat -an'. If I tried to restart target and stmf service, but stmf service got stucked in online* state and would not start. Reboot was the only solution in my case, but as I said, latest 014 release is working OK (but then again, load got reduced).
>>>> 
>>>> All good info. Thanks!
>>>> 
>>>> -brian
>>>> 
>>>>> 
>>>>> Matej
>>>>> 
>>>>>> On 08 Jan 2016, at 17:50, Dave Pooser <dave-oo at pooserville.com> wrote:
>>>>>> 
>>>>>>>> On Jan 8, 2016, at 11:22 AM, Brian Hechinger <wonko at 4amlunch.net> wrote:
>>>>>>>> 
>>>>>>>> No, ZFS raid10
>>>>>>> 
>>>>>>> Saw the HW-RAID term, and got concerned.  That's what, raidz2 in ZFS-ese?
>>>>>> 
>>>>>> It's a zpool with multiple mirror vdevs.
>>>>>> 
>>>>>> -- 
>>>>>> Dave Pooser
>>>>>> Cat-Herder-in-Chief, Pooserville.com
>>>>>> 
>>>>>> 
>>>>>> _______________________________________________
>>>>>> OmniOS-discuss mailing list
>>>>>> OmniOS-discuss at lists.omniti.com
>>>>>> http://lists.omniti.com/mailman/listinfo/omnios-discuss
>>>>> 
>>>>> _______________________________________________
>>>>> OmniOS-discuss mailing list
>>>>> OmniOS-discuss at lists.omniti.com
>>>>> http://lists.omniti.com/mailman/listinfo/omnios-discuss
>>>> 
>>>> _______________________________________________
>>>> OmniOS-discuss mailing list
>>>> OmniOS-discuss at lists.omniti.com
>>>> http://lists.omniti.com/mailman/listinfo/omnios-discuss
>>> 
>> 
>