[OmniOS-discuss] device probe related command timeouts

[Joshua M. Clulow]

On 4 January 2017 at 12:29, John Barfield <john.barfield at bissinc.com> wrote:
> I’ve got a SAN that seems to be timing out on any hardware probing commands such as “format” or “diskinfo” although prtconf seems to work.
>
> Does anyone happen to have a dtrace one liner or maybe kstat command I can run to see why/what they’re hanging on?

I would start by running "pstack" with the pid of one of the stuck
processes.  That will give you the part of the user program which is
stuck.  Then, I would get the in-kernel state of the stuck threads;
e.g., looking at my bash process:

    asgard # echo $$
    45435
    asgard # ps -fp 45435
         UID   PID  PPID   C    STIME TTY         TIME CMD
        root 45435 45433   0 20:47:17 pts/3       0:00 -bash

    asgard # mdb -k
    Loading modules: [ unix genunix specfs ... ]
    > 0t45435::pid2proc | ::ps -f
    S    PID   PPID   PGID    SID    UID      FLAGS             ADDR NAME
    R  45435  45433  45435  45435      0 0x4a014000 ffffff1b14d33048 -bash
    > 0t45435::pid2proc | ::walk thread | ::findstack -v
    stack pointer for thread ffffff03f776c080: ffffff0011b57c10
    [ ffffff0011b57c10 _resume_from_idle+0x112() ]
      ffffff0011b57c40 swtch+0x141()
      ffffff0011b57cd0 cv_wait_sig_swap_core+0x1b9(ffffff1b14d33108, ...)
      ffffff0011b57cf0 cv_wait_sig_swap+0x17(ffffff1b14d33108, ...)
      ffffff0011b57da0 waitid+0x315(7, 0, ffffff0011b57e30, f)
      ffffff0011b57eb0 waitsys32+0x36(7, 0, 8047750, f)
      ffffff0011b57f10 sys_syscall32+0x123()


That might tell us where in the storage subsystem you're getting stuck.


Cheers.

-- 
Joshua M. Clulow
UNIX Admin/Developer
http://blog.sysmgr.org