[OmniOS-discuss] [zfs] FMD fails to run

[Schweiss, Chip]

The fault manager is starting now.  However, the disks still as show as
UNAVAIL when running zpool import.

# zpool import
   pool: hcp-arc01
     id: 11579406004081253836
  state: UNAVAIL
 status: One or more devices are missing from the system.
 action: The pool cannot be imported. Attach the missing
        devices and try again.
   see: http://illumos.org/msg/ZFS-8000-3C
 config:

        hcp-arc01                                            UNAVAIL
insufficient replicas
          raidz3-0                                           UNAVAIL
insufficient replicas
            c0t5000C50093E3BE87d0p0                          UNAVAIL
cannot open
            c0t5000C50086B52EABd0p0                          UNAVAIL
cannot open
            c0t5000C50093F046A7d0p0                          UNAVAIL
cannot open
            c0t5000C50093E3086Fd0p0                          UNAVAIL
cannot open
            c0t5000C50093E85C07d0p0                          UNAVAIL
cannot open
            c0t5000C50093E3BED3d0p0                          UNAVAIL
cannot open
            c0t5000C50093E39267d0p0                          UNAVAIL
cannot open
            c0t5000C50093E309DBd0p0                          UNAVAIL
cannot open
            c0t5000C50093E31407d0p0                          UNAVAIL
cannot open
            c0t5000C50093E3885Bd0p0                          UNAVAIL
cannot open
            c0t5000C50093E344D7d0p0                          UNAVAIL
cannot open
            c0t5000C50093E332AFd0p0                          UNAVAIL
cannot open
            c0t5000C50093F04A2Fd0p0                          UNAVAIL
cannot open
            c0t5000C50093F04763d0p0                          UNAVAIL
cannot open
            c0t5000C50086B5DCE3d0p0                          UNAVAIL
cannot open
            c0t5000C50086B5CD37d0p0                          UNAVAIL
cannot open
            c0t5000C50086B5E263d0p0                          UNAVAIL
cannot open
            c0t5000C50086B5CD07d0p0                          UNAVAIL
cannot open
            c0t5000C50086B5DB3Bd0p0                          UNAVAIL
cannot open
            c0t5000C50086B5D95Fd0p0                          UNAVAIL
cannot open
            c0t5000C50086B566BBd0p0                          UNAVAIL
cannot open
            c0t5000C50086B5F38Fd0p0                          UNAVAIL
cannot open
            c0t5000C50093E37C97d0p0                          UNAVAIL
cannot open
            c0t5000C50093E3909Bd0p0                          UNAVAIL
cannot open
          raidz3-1                                           UNAVAIL
insufficient replicas
            c0t5000C50093E85C1Fd0p0                          UNAVAIL
cannot open
            c0t5000C50093E3A29Fd0p0                          UNAVAIL
cannot open
            c0t5000C50093E342BFd0p0                          UNAVAIL
cannot open
            c0t5000C50093E359DFd0p0                          UNAVAIL
cannot open
            c0t5000C50086B5281Fd0p0                          UNAVAIL
cannot open
            c0t5000C50093E331F7d0p0                          UNAVAIL
cannot open
            c0t5000C50093E35A93d0p0                          UNAVAIL
cannot open
            c0t5000C50093E38347d0p0                          UNAVAIL
cannot open
            c0t5000C50093E8532Bd0p0                          UNAVAIL
cannot open
            c0t5000C50093E3422Fd0p0                          UNAVAIL
cannot open
            c0t5000C50093CFA493d0p0                          UNAVAIL
cannot open
            c0t5000C50093E29DB3d0p0                          UNAVAIL
cannot open
            c0t5000C50093E3B70Bd0p0                          UNAVAIL
cannot open
            c0t5000C50093E3946Fd0p0                          UNAVAIL
cannot open
            c0t5000C50086B5319Bd0p0                          UNAVAIL
cannot open
            c0t5000C50086B5608Bd0p0                          UNAVAIL
cannot open
            c0t5000C50086B5D9B7d0p0                          UNAVAIL
cannot open
            c0t5000C50086B5E1ABd0p0                          UNAVAIL
cannot open
            c0t5000C50093E85D93d0p0                          UNAVAIL
cannot open
            c0t5000C50093E85C73d0p0                          UNAVAIL
cannot open
            c0t5000C50086B5D7CBd0p0                          UNAVAIL
cannot open
            c0t5000C50093E33F23d0p0                          UNAVAIL
cannot open
            c0t5000C50093E36A8Fd0p0                          UNAVAIL
cannot open
            c0t5000C50093E30193d0p0                          UNAVAIL
cannot open
          raidz3-2                                           UNAVAIL
insufficient replicas
            c0t5000C50093E34E3Fd0p0                          UNAVAIL
cannot open
            c0t5000C50093E36DB7d0p0                          UNAVAIL
cannot open
            c0t5000C50093E2C467d0p0                          UNAVAIL
cannot open
            c0t5000C50093E3A213d0p0                          UNAVAIL
cannot open
            c0t5000C50093E387E3d0p0                          UNAVAIL
cannot open
            c0t5000C50093E36DDFd0p0                          UNAVAIL
cannot open
            c0t5000C50093E852BBd0p0                          UNAVAIL
cannot open
            c0t5000C50093E85E5Bd0p0                          UNAVAIL
cannot open
            c0t5000C50093E85CB7d0p0                          UNAVAIL
cannot open
            c0t5000C50093E85CA3d0p0                          UNAVAIL
cannot open
            c0t5000C50093E85B77d0p0                          UNAVAIL
cannot open
            c0t5000C50093E85D6Fd0p0                          UNAVAIL
cannot open
            c0t5000C50093E85F83d0p0                          UNAVAIL
cannot open
            c0t5000C50093E85753d0p0                          UNAVAIL
cannot open
            c0t5000C50093E92BBBd0p0                          UNAVAIL
cannot open
            c0t5000C50093F04C73d0p0                          UNAVAIL
cannot open
            c0t5000C50093F04BEFd0p0                          UNAVAIL
cannot open
            c0t5000C50093F04C4Bd0p0                          UNAVAIL
cannot open
            c0t5000C50093F04163d0p0                          UNAVAIL
cannot open
            c0t5000C50093E8581Fd0p0                          UNAVAIL
cannot open
            c0t5000C50086B54183d0p0                          UNAVAIL
cannot open
            c0t5000C50093F04C6Fd0p0                          UNAVAIL
cannot open
            c0t5000C50093F0489Fd0p0                          UNAVAIL
cannot open
            c0t5000C50093E312A3d0p0                          UNAVAIL
cannot open
          raidz3-3                                           UNAVAIL
insufficient replicas
            c0t5000C50093E85E43d0p0                          UNAVAIL
cannot open
            c0t5000C50093E310A7d0p0                          UNAVAIL
cannot open
            c0t5000C50086B548DFd0p0                          UNAVAIL
cannot open
            c0t5000C50086B5EBFFd0p0                          UNAVAIL
cannot open
            c0t5000C50086B5B23Fd0p0                          UNAVAIL
cannot open
            c0t5000C50093E8579Fd0p0                          UNAVAIL
cannot open
            c0t5000C50093E343DFd0p0                          UNAVAIL
cannot open
            c0t5000C50093E3A103d0p0                          UNAVAIL
cannot open
            c0t5000C50086B528ABd0p0                          UNAVAIL
cannot open
            c0t5000C50093E85727d0p0                          UNAVAIL
cannot open
            c0t5000C50093F2FE4Bd0p0                          UNAVAIL
cannot open
            c0t5000C50093F045E7d0p0                          UNAVAIL
cannot open
            c0t5000C50093D9298Bd0p0                          UNAVAIL
cannot open
            c0t5000C50093F3141Fd0p0                          UNAVAIL
cannot open
            c0t5000C50093F04A53d0p0                          UNAVAIL
cannot open
            /dev/disk/by-id/dm-uuid-mpath-35000c50093f046b3  UNAVAIL
corrupted data
            c0t5000C50093F04853d0p0                          UNAVAIL
cannot open
            c0t5000C50093E85C0Bd0p0                          UNAVAIL
cannot open
            c0t5000C50093E3A9D3d0p0                          UNAVAIL
cannot open
            c0t5000C50093E31DEBd0p0                          UNAVAIL
cannot open
            c0t5000C50093F04A43d0p0                          UNAVAIL
cannot open
            c0t5000C50093F04C53d0p0                          UNAVAIL
cannot open
            c0t5000C50093F3009Fd0p0                          UNAVAIL
cannot open
            c0t5000C50093F04BDBd0p0                          UNAVAIL
cannot open


The "corrupted data" disk was kicked because of a predictive failure being
triggered:

# fmadm faulty
--------------- ------------------------------------  --------------
---------
TIME            EVENT-ID                              MSG-ID
 SEVERITY
--------------- ------------------------------------  --------------
---------
Mar 19 11:17:17 cd4beda0-c3c0-e335-b4a6-89887bc6966a  DISK-8000-0X   Major

Host        : hcp-raw-zfs01
Platform    : SYS-1018R-WC0R    Chassis_id  : S16229126215836
Product_sn  :

Fault class : fault.io.disk.predictive-failure
Affects     : dev:////scsi_vhci/disk@g5000c50093f046b3
                  faulted and taken out of service
FRU         : "SLOT 88 76"
(hc://:product-id=RAIDINC-96BAY:server-id=:chassis-id=500093d230938000:serial=ZA17RWNK0000R740EU4Z:part=SEAGATE-ST8000NM0075:revision=E004/ses-enclosure=0/bay=88/disk=0)
                  faulty

Description : SMART health-monitoring firmware reported that a disk
              failure is imminent.
              Refer to http://illumos.org/msg/DISK-8000-0X for more
              information.

Response    : None.

Impact      : It is likely that the continued operation of
              this disk will result in data loss.

Action      : Schedule a repair procedure to replace the affected disk.
              Use fmdump -v -u <EVENT_ID> to identify the disk.


This pool was created on Linux directly on the disk.   Could it be a
partitioning problem here?   Any suggestions how to fix this?

-Chip

On Mon, Mar 19, 2018 at 10:33 AM, Andy Fiddaman <omnios at citrus-it.net>
wrote:

>
> Chip,
>
> Try this hot-fix:
>
> # pkg apply-hot-fix --be-name=9317 \
>     https://downloads.omniosce.org/pkg/r151024/9317-ses2.p5p
>
> If it fixes your problem I'll work on upstreaming this fix to illumos-gate
> and it will definitely be in the upcoming OmniOS r151026 as well as
> being backported to r151022 & r151024.
>
> Regards,
>
> Andy
>
> --
> Citrus IT Limited | +44 (0)333 0124 007 | enquiries at citrus-it.co.uk
> Rock House Farm | Green Moor | Wortley | Sheffield | S35 7DQ
> Registered in England and Wales | Company number 4899123
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://omniosce.org/ml-archive/attachments/20180319/69a16a87/attachment-0001.html>