[OmniOS-discuss] 4kn or 512e with ashift=12
Fred Liu
Fred_Liu at issi.com
Thu Mar 24 10:26:20 UTC 2016
> -----Original Message-----
> From: Chris Siebenmann [mailto:cks at cs.toronto.edu]
> Sent: 星期三, 三月 23, 2016 23:33
> To: Richard Jahnel
> Cc: Chris Siebenmann; Fred Liu; omnios-discuss at lists.omniti.com
> Subject: Re: [OmniOS-discuss] 4kn or 512e with ashift=12
>
> > It should be noted that using a 512e disk as a 512n disk subjects you
> > to a significant risk of silent corruption in the event of power loss.
> > Because 512e disks does a read>modify>write operation to modify
> > 512byte chunk of a 4k sector, zfs won't know about the other
> > 7 corrupted 512e sectors in the event of a power loss during a write
> > operation. So when discards the incomplete txg on reboot, it won't do
> > anything about the other 7 512e sectors it doesn't know were affected.
>
> This is true; under normal circumstances you do not want to use a 512e drive
> in an ashift=9 vdev. However, if you have a dead 512n drive and you have no
> remaining 512n spares, your choices are to run without redundancy, to wedge
> in a 512e drive and accept the potential problems on power failure (problems
> that can likely be fixed by scrubbing the pool afterwards), or obtain enough
> additional drives (and perhaps
> server(s)) to entirely rebuild the pool on 512e drives with ashift=12.
>
> In this situation, running with a 512e drive and accepting the performance
> issues and potential exposure to power failures is basically the lesser evil. (I
> wish ZFS was willing to accept this, but it isn't.)
>
[Fred Liu]: I have a similar test here:
[root at 00-25-90-74-f5-04 ~]# zpool status
pool: tank
state: ONLINE
scan: resilvered 187G in 21h9m with 0 errors on Thu Jan 15 08:05:16 2015
config:
NAME STATE READ WRITE CKSUM
tank ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
c2t45d0 ONLINE 0 0 0
c2t46d0 ONLINE 0 0 0
c2t47d0 ONLINE 0 0 0
c2t48d0 ONLINE 0 0 0
c2t49d0 ONLINE 0 0 0
c2t52d0 ONLINE 0 0 0
c2t53d0 ONLINE 0 0 0
c2t44d0 ONLINE 0 0 0
spares
c0t5000CCA6A0C791CBd0 AVAIL
errors: No known data errors
pool: zones
state: ONLINE
scan: scrub repaired 0 in 2h45m with 0 errors on Tue Aug 12 20:24:30 2014
config:
NAME STATE READ WRITE CKSUM
zones ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
c0t5000C500584AC07Bd0 ONLINE 0 0 0
c0t5000C500584AC557d0 ONLINE 0 0 0
c0t5000C500584ACB1Fd0 ONLINE 0 0 0
c0t5000C500584AD7B3d0 ONLINE 0 0 0
c0t5000C500584C30DBd0 ONLINE 0 0 0
c0t5000C500586E54A3d0 ONLINE 0 0 0
c0t5000C500586EF0CBd0 ONLINE 0 0 0
c0t5000C50058426A0Fd0 ONLINE 0 0 0
logs
c4t0d0 ONLINE 0 0 0
c4t1d0 ONLINE 0 0 0
cache
c0t55CD2E404BE9CB7Ed0 ONLINE 0 0 0
errors: No known data errors
[root at 00-25-90-74-f5-04 ~]# format
Searching for disks...done
AVAILABLE DISK SELECTIONS:
0. c0t55CD2E404BE9CB7Ed0 <ATA-INTEL SSDSC2BW18-DC32-167.68GB>
/scsi_vhci/disk at g55cd2e404be9cb7e
1. c0t5000C500584AC07Bd0 <SEAGATE-ST91000640SS-0004-931.51GB>
/scsi_vhci/disk at g5000c500584ac07b
2. c0t5000C500584AC557d0 <SEAGATE-ST91000640SS-0004-931.51GB>
/scsi_vhci/disk at g5000c500584ac557
3. c0t5000C500584ACB1Fd0 <SEAGATE-ST91000640SS-0004-931.51GB>
/scsi_vhci/disk at g5000c500584acb1f
4. c0t5000C500584AD7B3d0 <SEAGATE-ST91000640SS-0004-931.51GB>
/scsi_vhci/disk at g5000c500584ad7b3
5. c0t5000C500584C30DBd0 <SEAGATE-ST91000640SS-0004-931.51GB>
/scsi_vhci/disk at g5000c500584c30db
6. c0t5000C500586E54A3d0 <SEAGATE-ST91000640SS-0004-931.51GB>
/scsi_vhci/disk at g5000c500586e54a3
7. c0t5000C500586EF0CBd0 <SEAGATE-ST91000640SS-0004-931.51GB>
/scsi_vhci/disk at g5000c500586ef0cb
8. c0t5000C50058426A0Fd0 <SEAGATE-ST91000640SS-0004-931.51GB>
/scsi_vhci/disk at g5000c50058426a0f
9. c0t5000CCA6A0C791CBd0 <ATA-Hitachi HTS54101-A480-931.51GB>
/scsi_vhci/disk at g5000cca6a0c791cb
10. c0t50000F0056425331d0 <ATA-SAMSUNG MMCRE28G-AS1Q-119.24GB>
/scsi_vhci/disk at g50000f0056425331
11. c2t44d0 <ATA-Hitachi HTS54101-A480-931.51GB>
/pci at 0,0/pci8086,1c10 at 1c/pci1000,3140 at 0/sd at 2c,0
12. c2t45d0 <ATA-Hitachi HTS54101-A480-931.51GB>
/pci at 0,0/pci8086,1c10 at 1c/pci1000,3140 at 0/sd at 2d,0
13. c2t46d0 <ATA-ST1000LM024 HN-M-0002-931.51GB>
/pci at 0,0/pci8086,1c10 at 1c/pci1000,3140 at 0/sd at 2e,0
14. c2t47d0 <ATA-ST1000LM024 HN-M-0002-931.51GB>
/pci at 0,0/pci8086,1c10 at 1c/pci1000,3140 at 0/sd at 2f,0
15. c2t48d0 <ATA-WDC WD10JPVT-08A-1A01-931.51GB>
/pci at 0,0/pci8086,1c10 at 1c/pci1000,3140 at 0/sd at 30,0
16. c2t49d0 <ATA-WDC WD10JPVT-75A-1A01-931.51GB>
/pci at 0,0/pci8086,1c10 at 1c/pci1000,3140 at 0/sd at 31,0
17. c2t52d0 <ATA-ST1000LM024 HN-M-0001-931.51GB>
/pci at 0,0/pci8086,1c10 at 1c/pci1000,3140 at 0/sd at 34,0
18. c2t53d0 <ATA-ST1000LM024 HN-M-0001-931.51GB>
/pci at 0,0/pci8086,1c10 at 1c/pci1000,3140 at 0/sd at 35,0
19. c4t0d0 <ATA-ANS9010_2NNN2NNN-_200-1.78GB>
/pci at 0,0/pci15d9,624 at 1f,2/disk at 0,0
20. c4t1d0 <ATA-ANS9010_2NNN2NNN-_200-1.78GB>
/pci at 0,0/pci15d9,624 at 1f,2/disk at 1,0
[root at 00-25-90-74-f5-04 ~]# zpool replace tank c2t44d0 c0t5000CCA6A0C791CBd0
cannot replace c2t44d0 with c0t5000CCA6A0C791CBd0: devices have different sector alignment
But in fact "c2t44d0" and "c0t5000CCA6A0C791CBd0" are in the same model -- ATA-Hitachi HTS54101-A480-931.51GB.
That is HTS541010A9E680 (https://www.hgst.com/sites/default/files/resources/TS5K1000_ds.pdf) which is a 512e HDD.
The *only* difference is that "c2t44d0" is attached to a LSI 1068 HBA and "c0t5000CCA6A0C791CBd0" is attached to a LSI 2308 HBA.
[root at 00-25-90-74-f5-04 ~]# zdb -l /dev/dsk/c2t44d0s0 | grep ashift
ashift: 9
ashift: 9
ashift: 9
ashift: 9
[root at 00-25-90-74-f5-04 ~]# zdb -l /dev/dsk/c0t5000CCA6A0C791CBd0s0 | grep ashift
ashift: 12
ashift: 12
ashift: 12
ashift: 12
format> inq
Vendor: ATA
Product: Hitachi HTS54101
Revision: A480
format> q
adding " "ATA Hitachi HTS54101", "physical-block-size:512"," into sd.conf
[root at 00-25-90-74-f5-04 ~]# update_drv -vf sd
Cannot unload module: sd
Will be unloaded upon reboot.
Forcing update of sd.conf.
sd.conf updated in the kernel.
Reboot the server for "cfgadm -c unconfigure" can't work here.
[root at 00-25-90-74-f5-04 ~]# zdb -l /dev/dsk/c0t5000CCA6A0C791CBd0s0 | grep ashift
[root at 00-25-90-74-f5-04 ~]#
No ashift in output now.
[root at 00-25-90-74-f5-04 ~]# zdb -l /dev/dsk/c2t44d0s0 | grep ashift
ashift: 9
ashift: 9
ashift: 9
ashift: 9
same like before
[root at 00-25-90-74-f5-04 ~]# zpool replace tank c2t44d0 c0t5000CCA6A0C791CBd0
cannot replace c2t44d0 with c0t5000CCA6A0C791CBd0: devices have different sector alignment
Remove the spare:
[root at 00-25-90-74-f5-04 ~]# zpool remove tank c0t5000CCA6A0C791CBd0
[root at 00-25-90-74-f5-04 ~]#
Add it back:
[root at 00-25-90-74-f5-04 ~]# zpool add tank spare c0t5000CCA6A0C791CBd0
[root at 00-25-90-74-f5-04 ~]#
[root at 00-25-90-74-f5-04 ~]# zpool status tank
pool: tank
state: ONLINE
scan: resilvered 187G in 21h9m with 0 errors on Thu Jan 15 08:05:16 2015
config:
NAME STATE READ WRITE CKSUM
tank ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
c2t45d0 ONLINE 0 0 0
c2t46d0 ONLINE 0 0 0
c2t47d0 ONLINE 0 0 0
c2t48d0 ONLINE 0 0 0
c2t49d0 ONLINE 0 0 0
c2t52d0 ONLINE 0 0 0
c2t53d0 ONLINE 0 0 0
c2t44d0 ONLINE 0 0 0
spares
c0t5000CCA6A0C791CBd0 AVAIL
errors: No known data errors
Still not working;
[root at 00-25-90-74-f5-04 ~]# zpool replace tank c2t44d0 c0t5000CCA6A0C791CBd0
cannot replace c2t44d0 with c0t5000CCA6A0C791CBd0: devices have different sector alignment
maybe the sd.conf update is not correct.
More information about the OmniOS-discuss
mailing list