[OmniOS-discuss] kernel panic

[Kevin Swab]

I've got OmniOS 151008j running on a home file server, and the other day
it went into a reboot loop, displaying a kernel panic on the console
just after the kernel banner was printed.

The panic message on screen showed some zfs function calls so following
that lead, I booted off the install media, mounted my root pool and
removed /etc/zpool.cache.  The system was able to boot after that but
when I attempt to import the pool containing my data, it panics again.

FMD shows that a reboot occurred after a kernel panic, and says more
info is available from fmdump.  Here's the stack trace from 'fmdump':

# fmdump -Vp -u 38f6aa49-6c97-4675-b526-e455b1ae215b
TIME                           UUID
SUNW-MSG-ID
Apr 07 2014 21:03:45.097921000 38f6aa49-6c97-4675-b526-e455b1ae215b
SUNOS-8000-KL

  TIME                 CLASS                                 ENA
  Apr 07 21:03:45.0237 ireport.os.sunos.panic.dump_available
0x0000000000000000
  Apr 07 21:03:03.8496 ireport.os.sunos.panic.dump_pending_on_device
0x0000000000000000

nvlist version: 0
        version = 0x0
        class = list.suspect
        uuid = 38f6aa49-6c97-4675-b526-e455b1ae215b
        code = SUNOS-8000-KL
        diag-time = 1396926225 62791
        de = fmd:///module/software-diagnosis
        fault-list-sz = 0x1
        fault-list = (array of embedded nvlists)
        (start fault-list[0])
        nvlist version: 0
                version = 0x0
                class = defect.sunos.kernel.panic
                certainty = 0x64
                asru =
sw:///:path=/var/crash/unknown/.38f6aa49-6c97-4675-b526-e455b1ae215b
                resource =
sw:///:path=/var/crash/unknown/.38f6aa49-6c97-4675-b526-e455b1ae215b
                savecore-succcess = 1
                dump-dir = /var/crash/unknown
                dump-files = vmdump.1
                os-instance-uuid = 38f6aa49-6c97-4675-b526-e455b1ae215b
                panicstr = BAD TRAP: type=e (#pf Page fault)
rp=ffffff000fadafc0 addr=2b8 occurred in module "unix" due to a NULL
pointer dereference
                panicstack = unix:die+df () | unix:trap+db3 () |
unix:cmntrap+e6 () | unix:mutex_enter+b () | zfs:zio_buf_alloc+25 () |
zfs:arc_get_data_buf+2b8 () | zfs:arc_buf_alloc+b5 () | zfs:arc_read+42b
() | zfs:dsl_scan_prefetch+a7 () | zfs:dsl_scan_recurse+16f () |
zfs:dsl_scan_visitbp+eb () | zfs:dsl_scan_visitdnode+bd () |
zfs:dsl_scan_recurse+439 () | zfs:dsl_scan_visitbp+eb () |
zfs:dsl_scan_visit_rootbp+61 () | zfs:dsl_scan_visit+26b () |
zfs:dsl_scan_sync+12f () | zfs:spa_sync+334 () | zfs:txg_sync_thread+227
() | unix:thread_start+8 () |
                crashtime = 1396801998
                panic-time = Sun Apr  6 10:33:18 2014 MDT
        (end fault-list[0])

        fault-status = 0x1
        severity = Major
        __ttl = 0x1
        __tod = 0x53436711 0x5d627e8



I'd really like to recover the data on that pool if possible, any
suggestions on what I can try next?

Thanks,
Kevin