[OmniOS-discuss] mixing 512B and 4K disks in vdev

[Chris Siebenmann]

> I have to replace a HDD disk in a mirrored vdev which is using
> ashift=9.  The problem is that all new HDD's is 4K disks so I wonder
> if anyone has made some performance measurements of using a 4K disk
> with ashift=9 in contrast to the obvious ashift=12?
>
> As I understand it you cannot detach a vdev from a pool so it is not
> possible to replace both disks with 4K disks. How do you handle a
> situation like this?

 OmniOS will not normally allow you to add a 4k disk to a vdev with
ashift=9, regardless of any performance impact. As far as I know, in
order to get around this you must modify /kernel/drv/sd.conf in order
to lie about the disk's physical sector size, per:

    http://wiki.illumos.org/display/illumos/ZFS+and+Advanced+Format+disks

Note that running 4k disks as 512-byte disks is ever so slightly risky,
because it converts what ZFS thinks are atomic writes of a sector into
read-modify-writes that affect some surrounding sectors.

 If you can't run 4k disks this way (or don't want to), as far as I know
your only option is to create an entire new pool with new vdevs that are
all ashift=12.

(This is where I really wish that all new vdevs were created with
ashift=12 as the default and you had to explicitly override it to get
ashift=9 vdevs, even on 512b drives. Creating ashift=9 vdevs is simply
not future proof at this point, and hasn't been for several years.)

	- cks