[OmniOS-discuss] Slow scrub on SSD-only pool

Thu Apr 21 10:41:55 UTC 2016

Am 19.04.16 um 23:31 schrieb wuffers:
> You might want to check this old thread:
>
> http://lists.omniti.com/pipermail/omnios-discuss/2014-July/002927.html
>
> Richard Elling had some interesting insights on how the scrub works:
>
> "So I think the pool is not scheduling scrub I/Os very well. You can 
> increase the number of
> scrub I/Os in the scheduler by adjusting the zfs_vdev_scrub_max_active 
> tunable. The
> default is 2, but you'll have to consider that a share (in the stock 
> market sense) where
> the active sync reads and writes are getting 10 each. You can try 
> bumping up the value
> and see what happens over some time, perhaps 10 minutes or so -- too 
> short of a time
> and you won't get a good feeling for the impact (try this in off-peak 
> time).
> echo zfs_vdev_scrub_max_active/W0t5 | mdb -kw
> will change the value from 2 to 5, increasing its share of the total 
> I/O workload.
>
> You can see the progress of scan (scrubs do scan) workload by looking 
> at the ZFS
> debug messages.
> echo ::zfs_dbgmsg | mdb -k
> These will look mysterious... they are. But the interesting bits are 
> about how many blocks
> are visited in some amount of time (txg sync interval). Ideally, this 
> will change as you
> adjust zfs_vdev_scrub_max_active."
>
> I had to increase my zfs_vdev_scrub_max_active parameter higher than 
> 5, but it sounds like the default setting for that tunable is no 
> longer satisfactory for today's high performance systems.
>
> On Sun, Apr 17, 2016 at 4:07 PM, Stephan Budach <stephan.budach at jvm.de 
> <mailto:stephan.budach at jvm.de>> wrote:
>
>     Am 17.04.16 um 20:42 schrieb Dale Ghent:
>
>             On Apr 17, 2016, at 9:07 AM, Stephan Budach
>             <stephan.budach at JVM.DE <mailto:stephan.budach at JVM.DE>> wrote:
>
>             Well… searching the net somewhat more thoroughfully, I
>             came across an archived discussion which deals also with a
>             similar issue. Somewhere down the conversation, this
>             parameter got suggested:
>
>             echo "zfs_scrub_delay/W0" | mdb -kw
>
>             I just tried that as well and although the caculated speed
>             climbs rathet slowly up, iostat now shows approx. 380 MB/s
>             read from the devices, which rates at 24 MB/s per single
>             device * 8 *2.
>
>             Being curious, I issued a echo "zfs_scrub_delay/W1" | mdb
>             -kw to see what would happen and that command immediately
>             drowned the rate on each device down to 1.4 MB/s…
>
>             What is the rational behind that? Who wants to wait for
>             weeks for a scrub to finish? Usually, I am having znapzend
>             run as well, creating snapshots on a regular basis.
>             Wouldn't that hurt scrub performance even more?
>
>         zfs_scrub_delay is described here:
>
>         http://src.illumos.org/source/xref/illumos-gate/usr/src/uts/common/fs/zfs/dsl_scan.c#63
>
>         How busy are your disks if you subtract the IO caused by a
>         scrub? Are you doing these scrubs with your VMs causing normal
>         IO as well?
>
>         Scrubbing, overall, is treated as a background maintenance
>         process. As such, it is designed to not interfere with
>         "production IO" requests. It used to be that scrubs ran as
>         fast as disk IO and bus bandwidth would allow, which in turn
>         severely impacted the IO performance of running applications,
>         and in some cases this would cause problems for production or
>         user services.  The scrub delay setting which you've
>         discovered is the main governor of this scrub throttle
>         code[1], and by setting it to 0, you are effectively removing
>         the delay it imposes on itself to allow non-scrub/resilvering
>         IO requests to finish.
>
>         The solution in your case is specific to yourself and how you
>         operate your servers and services. Can you accept degraded
>         application IO while a scrub or resilver is running? Can you
>         not? Maybe only during certain times?
>
>         /dale
>
>         [1]
>         http://src.illumos.org/source/xref/illumos-gate/usr/src/uts/common/fs/zfs/dsl_scan.c#1841
>
>     I do get the notion if this, but if the increase from 0 to 1
>     reduces the throughput from 24Mb/s to 1MB/s, this seems way
>     overboard to me. Having to wait for a couple of hours when running
>     with 0 as opposed to days (up to 10) when running at 1  - on a 1.3
>     TB zpool - doesn't seem to be the right choice. If this tunable
>     offered some more room for choice, that would be great, but it
>     obviously doesn't.
>
>     It's the weekend and my VMs aren't excatly hogging their disks, so
>     there was plenty of I/O available… I'd wish for a more granular
>     setting regarding this setting.
>
>     Anyway, the scrub finished a couple of hours later and of course,
>     I can always set this tunable to 0, should I need it,
>
>     Thanks,
>
>     Stephan
>

Interesting read - and it surely works. If you set the tunable before 
you start the scrub you can immediately see the thoughput being much 
higher than with the standard setting. Now, what would be the tunable to 
use zfs_vdev_scrub_max_active or zfs_scrub_delay?

Cheers,
Stephan

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://omniosce.org/ml-archive/attachments/20160421/71c35ae2/attachment-0001.html>