[OmniOS-discuss] [developer] NVMe Performance

[Josh Coombs]

On Fri, Apr 15, 2016 at 9:26 PM, Richard Yao <ryao at gentoo.org> wrote:

>
> The first is to make sure that ZFS uses proper alignment on the device.
> According to what I learned via Google searches, the Intel DC P3600
> supports both 512-byte sectors and 4096-byte sectors, but is low leveled
> formatted to 512-byte sectors by default. You could run fio to see how the
> random IO performance differs on 512-byte IOs at 512-byte formatting vs 4KB
> IOs at 4KB formatting, but I expect that you will find it performs best in
> the 4KB case like Intel's enterprise SATA SSDs do. If the 512-byte random
> IO performance was notable, Intel would have advertised it, but they did
> not do that:
>
>
> http://www.intel.com/content/dam/www/public/us/en/documents/product-specifications/ssd-dc-p3600-spec.pdf
>
> http://www.cadalyst.com/%5Blevel-1-with-primary-path%5D/how-configure-oracle-redo-intel-pcie-ssd-dc-p3700-23534
>
> So, I played around with this.  Intel's isdct tool will let you secure
erase the P3600 and set it up as a 4k sector device, or a 512, with a few
other options as well.  I have to re-look but it might support 8k sectors
too.  Unfortunately the NVMe driver doesn't play well with the SSD
formatted for anything other than 512 byte sectors.  I noted my findings in
Illumos bug #6912.

I need to look at how Illumos partitions the devices if you just feed zpool
the device rather than a partition, I didn't look to see if it was aligning
things correctly or not on it's own.

The second is that it is possible to increase IOPS beyond Intel's
> specifications by doing a secure erase, giving SLOG a tiny 4KB aligned
> partition and leaving the rest of the device unused. Intel's numbers are
> for steady state performance where almost every flash page is dirty. If you
> leave a significant number of pages clean (i.e. unused following a secure
> erase), the drive should perform better than what Intel claims by virtue of
> the internal book keeping and garbage collection having to do less. Anandtech
> has benchmarks numbers showing this effect on older consumer SSDs on
> Windows in a comparison with the Intel DC S3700:
>

Using isdct I have mine set to 50% over-provisioning, so they show up as
200GB devices now.  As noted in bug 6912 you have to secure erase after
changing that setting or the NVMe driver REALLY gets unhappy.

Josh C
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://omniosce.org/ml-archive/attachments/20160415/4a51164f/attachment.html>