[OmniOS-discuss] kernel panic

[Dan McDonald]

> On Sep 19, 2016, at 11:51 AM, Michael Rasmussen <mir at miras.org> wrote:
> 
> On Mon, 19 Sep 2016 11:35:33 -0400
> Dan McDonald <danmcd at omniti.com> wrote:
> 
>> 
>> I see you have a vmdump.2 present.  Can you make that downloadable somewhere? I'd like to take a closer look.
>> 
> ftp://ftp.datanom.net/pub/vmdump.2

This process:

R  15776   2671     10     10      0 0x4a004000 ffffff075dadd0a0 perl
        T  0xffffff072006ec60 <TS_SLEEP>

Hit a page fault:

> 0xffffff071b9a8400::findstack -v
stack pointer for thread ffffff071b9a8400: ffffff00206eb1f0
  ffffff00206eb290 page_ctr_sub_internal+0x65(0, 3, ffffff0012581270, 1)
  ffffff00206eb2f0 page_ctr_sub+0x7e(0, 3, ffffff0012581270, 1)
  ffffff00206eb410 page_get_mnode_freelist+0x3ff(0, 4, 3, 0, 202000b)
  ffffff00206eb4e0 page_get_freelist+0x16d(286, 0, baddcafebaddcafe, 
  fffffffffb934d9f, ffffff00206eb510, ffffff06e6fd8d00, ffffffff00000008)
  ffffff00206eb550 param_preset()
  ffffff00206eb5e0 0xfffffffffb8480fc()
  ffffff00206eb6f0 trap+0xa30(ffffff00206eb700, ffffff06e6fd8d00, 2)
  ffffff00206eb700 0xfffffffffb8001d6()
  ffffff00206eb8a0 fop_getpage+0x3f(ffffff06e6fd8d00, 1fffffe0ed7a1000, 1000, 
  ffffff00206eba90, ffffff00206eb9f0, 1000, ffffff0760c25938, 8178000, 
  ffffff0000000002, ffffff071db23ad8, 0)
  ffffff00206eb960 anon_getpage+0x18d(ffffff00206eba08, ffffff00206eba90, 
  ffffff00206eb9f0, 1000, ffffff0760c25938, 8178000, 2, ffffff071db23ad8)
  ffffff00206eba80 segvn_faultpage+0x168(ffffff07213e2020, ffffff0760c25938, 
  8178000, 116000, 0, ffffff00206ebb50, 0, 1, 2, 1)
  ffffff00206ebc50 segvn_fault+0x8e6(ffffff07213e2020, ffffff0760c25938, 8178000
  , 1000, 1, 2)
  ffffff00206ebd60 as_fault+0x31a(ffffff07213e2020, ffffff072245adc8, 817824c, 1
  , 1, 2)
  ffffff00206ebdf0 pagefault+0x96(817824c, 1, 2, 0)
  ffffff00206ebf00 trap+0x2c7(ffffff00206ebf10, 817824c, 2)
  ffffff00206ebf10 0xfffffffffb8001d6()
> 

And then trapped further while doing fop_getpage().  What's your swap device?

I also couldn't help but notice this:

> ffffff06e6fd8d00::print vnode_t v_vfsp
v_vfsp = 0xbaddcafebaddcafe
> 


WHOA!

> ffffff06e6fd8d00::whatis
ffffff06e6fd8d00 is ffffff06e6fd85c0+740, allocated from rootnex_dmahdl
> 


So fop_getpage() was sent a vnode_t that at least had one field uninitialized (0xbaddcafe).  Turns out the whole wad is:

> ffffff06e6fd8d00/10P
0xffffff06e6fd8d00:             0xbaddcafebaddcafe0xbaddcafebaddcafe
                0xbaddcafebaddcafe0xbaddcafebaddcafe0xbaddcafebaddcafe
                0xbaddcafebaddcafe0xbaddcafebaddcafe0xbaddcafebaddcafe
                0xbaddcafebaddcafe0xbaddcafebaddcafe
> 

The anon cache entry fed into anon_getpage seems legit, but corrupt somehow:

> ffffff00206eba08/P
0xffffff00206eba08:             0xffffff076bd0a5c0
> 0xffffff076bd0a5c0::print "struct anon"
{
    an_vp = 0xffffff06e6fd8d00
    an_pvp = 0
    an_off = 0x1fffffe0ed7a1000
    an_poff = 0
    an_hash = 0
    an_refcnt = 0x1
}
> 0xffffff06e6fd8d00::whatis
ffffff06e6fd8d00 is ffffff06e6fd85c0+740, allocated from rootnex_dmahdl
> 0xffffff06e6fd8d00/20P    
0xffffff06e6fd8d00:             0xbaddcafebaddcafe0xbaddcafebaddcafe
                0xbaddcafebaddcafe0xbaddcafebaddcafe0xbaddcafebaddcafe
                0xbaddcafebaddcafe0xbaddcafebaddcafe0xbaddcafebaddcafe
                0xbaddcafebaddcafe0xbaddcafebaddcafe0xbaddcafebaddcafe
                0xbaddcafebaddcafe0xbaddcafebaddcafe0xbaddcafebaddcafe
                0xbaddcafebaddcafe0xbaddcafebaddcafe0xbaddcafebaddcafe
                0xbaddcafebaddcafe0xbaddcafebaddcafe0xbaddcafebaddcafe
> 0xffffff076bd0a5c0::whatis
ffffff076bd0a5c0 is allocated from anon_cache
> 

Generally speaking, this is memory corruption.  What caused it I'm not sure, however.

Dan