[OmniOS-discuss] slow drive response times

[Richard Elling]

> On Dec 31, 2014, at 11:25 AM, Kevin Swab <Kevin.Swab at colostate.edu> wrote:
> 
> Hello Everyone,
> 
> 	We've been running OmniOS on a number of SuperMicro 36bay chassis, with
> Supermicro motherboards, LSI SAS controllers (9211-8i & 9207-8i) and
> various SAS HDD's.  These systems are serving block storage via Comstar
> and Qlogic FC HBA's, and have been running well for several years.
> 
> 	The problem we've got is that as the drives age, some of them start to
> perform slowly (intermittently) without failing - no zpool or iostat
> errors, and nothing logged in /var/adm/messages.  The slow performance
> can be seen as high average service times in iostat or sar.

Look at the drive's error logs using sg_logs (-a for all)

> 
> 	When these service times get above 500ms, they start to cause IO
> timeouts on the downstream storage consumers, which is bad...

500 milliseconds is not unusual for a busy HDD with SCSI TCQ or SATA NCQ

> 
> 	I'm wondering - is there a way to tune OmniOS' behavior so that it
> doesn't try so hard to complete IOs to these slow disks, and instead
> just gives up and fails them?

Yes, the tuning in Alasdair's blog should work as he describes. More below...

> 
> 	I found an old post from 2011 which states that some tunables exist,
> but are ignored by the mpt_sas driver:
> 
> http://everycity.co.uk/alasdair/2011/05/adjusting-drive-timeouts-with-mdb-on-solaris-or-openindiana/
> 
> Does anyone know the current status of these tunables, or have any other
> suggestions that might help?

These tunables are on the order of seconds. The default, 60, is obviously too big
unless you have old, slow, SCSI CD-ROMs. But setting it below the manufacturer's
internal limit (default or tuned) can lead to an unstable system. Some vendors are
better than others at documenting these, but in any case you'll need to see their spec.
Expect values on the order of 6 to 15 seconds for modern HDDs and SSDs.

There are a lot of tunables in this area at all levels of the architecture. OOB, the OmniOS
settings ensure stable behaviour. Tuning any layer without understanding the others can
lead to unstable systems, as demonstrated by your current downstream consumers.
 -- richard


> 
> Thanks,
> Kevin
> 
> 
> -- 
> -------------------------------------------------------------------
> Kevin Swab                          UNIX Systems Administrator
> ACNS                                Colorado State University
> Phone: (970)491-6572                Email: Kevin.Swab at ColoState.EDU
> GPG Fingerprint: 7026 3F66 A970 67BD 6F17  8EB8 8A7D 142F 2392 791C
> _______________________________________________
> OmniOS-discuss mailing list
> OmniOS-discuss at lists.omniti.com
> http://lists.omniti.com/mailman/listinfo/omnios-discuss