[OmniOS-discuss] Understanding OmniOS disk IO timeouts and options to control them

[Richard Elling]

one more thing…

> On Jan 4, 2017, at 10:29 AM, Richard Elling <richard.elling at richardelling.com> wrote:
> 
>> 
>> On Jan 4, 2017, at 10:04 AM, Chris Siebenmann <cks at cs.toronto.edu> wrote:
>> 
>> We recently had a server reboot due to the ZFS vdev_deadman/spa_deadman
>> timeout timer activating and panicing the system. If you haven't heard
>> of this timer before, that's not surprising; triggering it requires an
>> IO to a vdev to take more than 1000 seconds (by default; it's controlled
>> by the value of zfs_deadman_synctime_ms, in spa_misc.c).
>> 
>> Before this happened, I would not have expected that our OmniOS system
>> allowed an IO to run that long before timing it out and returning an
>> error to ZFS. Clearly I'm wrong, which means that I'd like to understand
>> what disk IO timeouts OmniOS has and where (or if) we can control them
>> so that really long IOs get turned into forced errors well before 16
>> minutes go by. Unfortunately our disk topology is a bit complicated;
>> we have scsi_vhci multipathing on top of iSCSI disks.
> 
> Do not assume the timeout reflects properly operating software or firmware.
> The original impetus for the deadman was to allow debugging of the underlying
> stack. Prior to adding the deadman, the I/O could be stuck forever.
> 
>> 
>> In some Internet searching I've found sd_io_time (60 seconds by
>> default) and the default SD retry count of 5 (I think, it may be
>> 3), which can be adjusted on a per-disk-type basis through the
>> retries-timeout parameter (per the sd manpage). Searching the kernel
>> code suggests that there are some hard-coded timeouts in the 30 to 90
>> second range, which also doesn't seem excessive.
> 
> For sd-level, most commands follow the sd_io_time and retries. scsi_vhci adds
> significant complexity above sd and below zfs.
> — richard
> 
>> 
>> (I have a crash dump from this panic, so I can in theory use mdb
>> to look through it to see just what level an IO appears stuck at
>> if I know what to look for and how.)
>> 
>> Based on 'fmdump -eV' output, it looks like our server was
>> retrying IO repeatedly.
>> 
>> Does anyone know what I should be looking at to find and adjust
>> timeouts, retry counts, and so on? Is there any documentation that
>> I'm overlooking?

I find the "zio_state" mdb macro helpful in these cases. It shows all of the I/Os in the zio pipeline
and the timeout values relative to the deadman.
 — richard

>> 
>> Thanks in advance.
>> 
>> 	- cks
>> PS: some links I've dug up in searches:
>>   http://everycity.co.uk/alasdair/2011/05/adjusting-drive-timeouts-with-mdb-on-solaris-or-openindiana/
>>   https://smartos.org/bugview/OS-2415
>>   https://www.illumos.org/issues/1553
>> _______________________________________________
>> OmniOS-discuss mailing list
>> OmniOS-discuss at lists.omniti.com
>> http://lists.omniti.com/mailman/listinfo/omnios-discuss

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://omniosce.org/ml-archive/attachments/20170104/64a95d4e/attachment-0001.html>