<html><head><meta http-equiv="Content-Type" content="text/html charset=windows-1252"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><br class=""><div><blockquote type="cite" class=""><div class="">On Jan 2, 2015, at 1:50 PM, Kevin Swab <<a href="mailto:Kevin.Swab@colostate.edu" class="">Kevin.Swab@colostate.edu</a>> wrote:</div><br class="Apple-interchange-newline"><div class="">I've run 'sg_logs' on the drive I pulled last week. There were alot of<br class="">errors in the backgroud scan section of the output, which made it very<br class="">large, so I put it here:<br class=""><br class=""><a href="http://pastebin.com/jx5BvSep" class="">http://pastebin.com/jx5BvSep</a><br class=""><br class="">When I pulled this drive, the SMART health status was OK. </div></blockquote><div><br class=""></div><div>SMART isn’t smart :-P</div><br class=""><blockquote type="cite" class=""><div class="">However, when<br class="">I put it in a test system to run 'sg_logs', the status changed to<br class="">"impending failure...". Had the SMART status changed before pulling the<br class="">drive, I'm sure 'fmd' would have alerted me to the problem…</div></blockquote><div><br class=""></div><div>By default, fmd looks for the predictive failure (PFA) and self-test every hour using the disk_transport </div><div>agent. fmstat should show activity there. When a PFA is seen, then there will be an ereport generated</div><div>and, for most cases, a syslog message. However, this will not cause a zfs-retire event.</div><div><br class=""></div><div>Vendors have significant leeway in how they implement SMART. In my experience the only thing</div><div>you can say for sure is if the vendor thinks the drive’s death is imminent, then you should replace</div><div>it. I suspect these policies are financially motivated rather than scientific… some amount of truthiness</div><div>is to be expected.</div><div><br class=""></div><div>In the logs, clearly the one disk has lots of errors that have been corrected and the rate is increasing.</div><div>The rate of change for "Errors corrected with possible delays” may correlate to your performance issues,</div><div>but the interpretation is left up to the vendors.</div><div><br class=""></div><div>In the case of this naughty drive, yep it needs replacing.</div><br class=""><blockquote type="cite" class=""><div class=""><br class="">Since that drive had other indications of trouble, I ran 'sg_logs' on<br class="">another drive I pulled recently that has a SMART health status of OK,<br class="">but exibits similar slow service time behavior:<br class=""><br class=""><a href="http://pastebin.com/Q0t8Jnug" class="">http://pastebin.com/Q0t8Jnug</a><br class=""></div></blockquote><div><br class=""></div><div>This one looks mostly healthy.</div><div><br class=""></div><div>Another place to look for latency issues is the phy logs. In the sg_logs output, this is the</div><div><div class="">Protocol Specific port log page for SAS SSP. Key values are running disparity error </div><div class="">count and loss of dword sync count. The trick here is that you need to look at both ends</div><div class="">of the wire for each wire. For a simple case, this means looking at both the HBA’s phys error</div><div class="">counts and the driver. If you have expanders in the mix, it is more work. You’ll want to look at </div><div class="">all of the HBA, expander, and drive phys health counters for all phys.</div><div class=""><br class=""></div><div class="">This can get tricky because wide ports are mostly dumb. For example, if an HBA has a 4-link</div><div class="">wide port (common) and one of the links is acting up (all too common) the latency impacts </div><div class="">will be random.</div><div class=""><br class=""></div><div class="">To see HBA and expander link health, you can use sg3_utils, its companion smp_utils, or</div><div class="">sasinfo (installed as a separate package from OmniOS, IIRC). For example, </div><div class=""><span class="Apple-tab-span" style="white-space:pre"> </span>sasinfo hba-port -l</div><div class=""><br class=""></div><div class="">HTH</div><div class=""> — richard</div><div class=""><br class=""></div></div><br class=""><blockquote type="cite" class=""><div class=""><br class="">Thanks for taking the time to look at these, please let me know what you<br class="">find...<br class=""><br class="">Kevin<br class=""><br class=""><br class=""><br class=""><br class="">On 12/31/2014 06:13 PM, Richard Elling wrote:<br class=""><blockquote type="cite" class=""><br class=""><blockquote type="cite" class="">On Dec 31, 2014, at 4:30 PM, Kevin Swab <<a href="mailto:Kevin.Swab@ColoState.EDU" class="">Kevin.Swab@ColoState.EDU</a>> wrote:<br class=""><br class="">Hello Richard and group, thanks for your reply!<br class=""><br class="">I'll look into sg_logs for one of these devices once I have a chance to<br class="">track that progam down...<br class=""><br class="">Thanks for the tip on the 500 ms latency, I wasn't aware that could<br class="">happen in normal cases. However, I don't believe what I'm seeing<br class="">constitutes normal behavior.<br class=""><br class="">First, some anecdotal evidence: If I pull and replace the suspect<br class="">drive, my downstream systems stop complaining, and the high service time<br class="">numbers go away.<br class=""></blockquote><br class="">We call these "wounded soldiers" -- it takes more resources to manage a<br class="">wounded soldier than a dead soldier, so one strategy of war is to wound your<br class="">enemy causing them to consume resources tending the wounded. The sg_logs<br class="">should be enlightening.<br class=""><br class="">NB, consider a 4TB disk with 5 platters: if a head or surface starts to go, then<br class="">you have a 1/10 chance that the data you request is under the damaged head<br class="">and will need to be recovered by the drive. So it is not uncommon to see<br class="">90+% of the I/Os to the drive completing quickly. It is also not unusual to see<br class="">only a small number of sectors or tracks affected.<br class=""><br class="">Detecting these becomes tricky, especially as you reduce the timeout/retry<br class="">interval, since the problem is rarely seen in the average latency -- that which<br class="">iostat and sar record. This is an area where we can and are improving.<br class=""> -- richard<br class=""><br class=""><blockquote type="cite" class=""><br class="">I threw out 500 ms as a guess to the point at which I start seeing<br class="">problems. However, I see service times far in excess of that, sometimes<br class="">over 30,000 ms! Below is 20 minutes of sar output from a drive I pulled<br class="">a few days ago, during a time when downstream VMWare servers were<br class="">complaining. (since the sar output is so verbose, I grepped out the<br class="">info just for the suspect drive):<br class=""><br class=""># sar -d -f /var/adm/sa/sa28 -s 14:50 -e 15:10 | egrep '(device|sd91,a)'<br class="">14:50:00 device %busy avque r+w/s blks/s avwait avserv<br class=""> sd91,a 99 5.3 1 42 0.0 7811.7<br class=""> sd91,a 100 11.3 1 53 0.0 11016.0<br class=""> sd91,a 100 3.8 1 75 0.0 3615.8<br class=""> sd91,a 100 4.9 1 25 0.0 8633.5<br class=""> sd91,a 93 3.9 1 55 0.0 4385.3<br class=""> sd91,a 86 3.5 2 75 0.0 2060.5<br class=""> sd91,a 91 3.1 4 80 0.0 823.8<br class=""> sd91,a 97 3.5 1 50 0.0 3984.5<br class=""> sd91,a 100 4.4 1 56 0.0 6068.6<br class=""> sd91,a 100 5.0 1 55 0.0 8836.0<br class=""> sd91,a 100 5.7 1 51 0.0 7939.6<br class=""> sd91,a 98 9.9 1 42 0.0 12526.8<br class=""> sd91,a 100 7.4 0 10 0.0 36813.6<br class=""> sd91,a 51 3.8 8 90 0.0 500.2<br class=""> sd91,a 88 3.4 1 60 0.0 2338.8<br class=""> sd91,a 100 4.5 1 28 0.0 6969.2<br class=""> sd91,a 93 3.8 1 59 0.0 5138.9<br class=""> sd91,a 79 3.1 1 59 0.0 3143.9<br class=""> sd91,a 99 4.7 1 52 0.0 5598.4<br class=""> sd91,a 100 4.8 1 62 0.0 6638.4<br class=""> sd91,a 94 5.0 1 54 0.0 3752.7<br class=""><br class="">For comparison, here's the sar output from another drive in the same<br class="">pool for the same period of time:<br class=""><br class=""># sar -d -f /var/adm/sa/sa28 -s 14:50 -e 15:10 | egrep '(device|sd82,a)'<br class="">14:50:00 device %busy avque r+w/s blks/s avwait avserv<br class=""> sd82,a 0 0.0 2 28 0.0 5.6<br class=""> sd82,a 1 0.0 3 51 0.0 5.4<br class=""> sd82,a 1 0.0 4 66 0.0 6.3<br class=""> sd82,a 1 0.0 3 48 0.0 4.3<br class=""> sd82,a 1 0.0 3 45 0.0 6.1<br class=""> sd82,a 1 0.0 6 82 0.0 2.7<br class=""> sd82,a 1 0.0 8 112 0.0 2.8<br class=""> sd82,a 0 0.0 3 27 0.0 1.8<br class=""> sd82,a 1 0.0 5 80 0.0 3.1<br class=""> sd82,a 0 0.0 3 35 0.0 3.1<br class=""> sd82,a 1 0.0 3 35 0.0 3.8<br class=""> sd82,a 1 0.0 4 49 0.0 3.2<br class=""> sd82,a 0 0.0 0 0 0.0 4.1<br class=""> sd82,a 3 0.0 9 84 0.0 4.1<br class=""> sd82,a 1 0.0 6 55 0.0 3.7<br class=""> sd82,a 0 0.0 1 23 0.0 7.0<br class=""> sd82,a 0 0.0 6 57 0.0 1.8<br class=""> sd82,a 1 0.0 5 70 0.0 2.3<br class=""> sd82,a 1 0.0 4 55 0.0 3.7<br class=""> sd82,a 1 0.0 5 72 0.0 4.1<br class=""> sd82,a 1 0.0 4 54 0.0 3.6<br class=""><br class="">The other drives in this pool all show data similar to that of sd82.<br class=""><br class="">Your point about tuning blindly is well taken, and I'm certainly no<br class="">expert on the IO stack. What's a humble sysadmin to do?<br class=""><br class="">For further reference, this system is running r151010. The drive in<br class="">question is a Toshiba MG03SCA300 (7200rpm SAS), and the pool the drive<br class="">was in is using lz4 compression and looks like this:<br class=""><br class=""># zpool status data1<br class=""> pool: data1<br class="">state: ONLINE<br class=""> scan: resilvered 1.67T in 70h56m with 0 errors on Wed Dec 31 14:40:20 2014<br class="">config:<br class=""><br class=""> NAME STATE READ WRITE CKSUM<br class=""> data1 ONLINE 0 0 0<br class=""> raidz2-0 ONLINE 0 0 0<br class=""> c6t5000039468CB54F0d0 ONLINE 0 0 0<br class=""> c6t5000039478CB5138d0 ONLINE 0 0 0<br class=""> c6t5000039468D000DCd0 ONLINE 0 0 0<br class=""> c6t5000039468D000E8d0 ONLINE 0 0 0<br class=""> c6t5000039468D00F5Cd0 ONLINE 0 0 0<br class=""> c6t5000039478C816CCd0 ONLINE 0 0 0<br class=""> c6t5000039478C8546Cd0 ONLINE 0 0 0<br class=""> raidz2-1 ONLINE 0 0 0<br class=""> c6t5000039478C855F0d0 ONLINE 0 0 0<br class=""> c6t5000039478C856E8d0 ONLINE 0 0 0<br class=""> c6t5000039478C856ECd0 ONLINE 0 0 0<br class=""> c6t5000039478C856F4d0 ONLINE 0 0 0<br class=""> c6t5000039478C86374d0 ONLINE 0 0 0<br class=""> c6t5000039478C8C2A8d0 ONLINE 0 0 0<br class=""> c6t5000039478C8C364d0 ONLINE 0 0 0<br class=""> raidz2-2 ONLINE 0 0 0<br class=""> c6t5000039478C9958Cd0 ONLINE 0 0 0<br class=""> c6t5000039478C995C4d0 ONLINE 0 0 0<br class=""> c6t5000039478C9DACCd0 ONLINE 0 0 0<br class=""> c6t5000039478C9DB30d0 ONLINE 0 0 0<br class=""> c6t5000039478C9DB6Cd0 ONLINE 0 0 0<br class=""> c6t5000039478CA73B4d0 ONLINE 0 0 0<br class=""> c6t5000039478CB3A20d0 ONLINE 0 0 0<br class=""> raidz2-3 ONLINE 0 0 0<br class=""> c6t5000039478CB3A64d0 ONLINE 0 0 0<br class=""> c6t5000039478CB3A70d0 ONLINE 0 0 0<br class=""> c6t5000039478CB3E7Cd0 ONLINE 0 0 0<br class=""> c6t5000039478CB3EB0d0 ONLINE 0 0 0<br class=""> c6t5000039478CB3FBCd0 ONLINE 0 0 0<br class=""> c6t5000039478CB4048d0 ONLINE 0 0 0<br class=""> c6t5000039478CB4054d0 ONLINE 0 0 0<br class=""> raidz2-4 ONLINE 0 0 0<br class=""> c6t5000039478CB424Cd0 ONLINE 0 0 0<br class=""> c6t5000039478CB4250d0 ONLINE 0 0 0<br class=""> c6t5000039478CB470Cd0 ONLINE 0 0 0<br class=""> c6t5000039478CB471Cd0 ONLINE 0 0 0<br class=""> c6t5000039478CB4E50d0 ONLINE 0 0 0<br class=""> c6t5000039478CB50A8d0 ONLINE 0 0 0<br class=""> c6t5000039478CB50BCd0 ONLINE 0 0 0<br class=""> spares<br class=""> c6t50000394A8CBC93Cd0 AVAIL<br class=""><br class="">errors: No known data errors<br class=""><br class=""><br class="">Thanks for your help,<br class="">Kevin<br class=""><br class="">On 12/31/2014 3:22 PM, Richard Elling wrote:<br class=""><blockquote type="cite" class=""><br class=""><blockquote type="cite" class="">On Dec 31, 2014, at 11:25 AM, Kevin Swab <<a href="mailto:Kevin.Swab@colostate.edu" class="">Kevin.Swab@colostate.edu</a>> wrote:<br class=""><br class="">Hello Everyone,<br class=""><br class=""><span class="Apple-tab-span" style="white-space:pre"> </span>We've been running OmniOS on a number of SuperMicro 36bay chassis, with<br class="">Supermicro motherboards, LSI SAS controllers (9211-8i & 9207-8i) and<br class="">various SAS HDD's. These systems are serving block storage via Comstar<br class="">and Qlogic FC HBA's, and have been running well for several years.<br class=""><br class=""><span class="Apple-tab-span" style="white-space:pre"> </span>The problem we've got is that as the drives age, some of them start to<br class="">perform slowly (intermittently) without failing - no zpool or iostat<br class="">errors, and nothing logged in /var/adm/messages. The slow performance<br class="">can be seen as high average service times in iostat or sar.<br class=""></blockquote><br class="">Look at the drive's error logs using sg_logs (-a for all)<br class=""><br class=""><blockquote type="cite" class=""><br class=""><span class="Apple-tab-span" style="white-space:pre"> </span>When these service times get above 500ms, they start to cause IO<br class="">timeouts on the downstream storage consumers, which is bad...<br class=""></blockquote><br class="">500 milliseconds is not unusual for a busy HDD with SCSI TCQ or SATA NCQ<br class=""><br class=""><blockquote type="cite" class=""><br class=""><span class="Apple-tab-span" style="white-space:pre"> </span>I'm wondering - is there a way to tune OmniOS' behavior so that it<br class="">doesn't try so hard to complete IOs to these slow disks, and instead<br class="">just gives up and fails them?<br class=""></blockquote><br class="">Yes, the tuning in Alasdair's blog should work as he describes. More below...<br class=""><br class=""><blockquote type="cite" class=""><br class=""><span class="Apple-tab-span" style="white-space:pre"> </span>I found an old post from 2011 which states that some tunables exist,<br class="">but are ignored by the mpt_sas driver:<br class=""><br class=""><a href="http://everycity.co.uk/alasdair/2011/05/adjusting-drive-timeouts-with-mdb-on-solaris-or-openindiana/" class="">http://everycity.co.uk/alasdair/2011/05/adjusting-drive-timeouts-with-mdb-on-solaris-or-openindiana/</a><br class=""><br class="">Does anyone know the current status of these tunables, or have any other<br class="">suggestions that might help?<br class=""></blockquote><br class="">These tunables are on the order of seconds. The default, 60, is obviously too big<br class="">unless you have old, slow, SCSI CD-ROMs. But setting it below the manufacturer's<br class="">internal limit (default or tuned) can lead to an unstable system. Some vendors are<br class="">better than others at documenting these, but in any case you'll need to see their spec.<br class="">Expect values on the order of 6 to 15 seconds for modern HDDs and SSDs.<br class=""><br class="">There are a lot of tunables in this area at all levels of the architecture. OOB, the OmniOS<br class="">settings ensure stable behaviour. Tuning any layer without understanding the others can<br class="">lead to unstable systems, as demonstrated by your current downstream consumers.<br class="">-- richard<br class=""><br class=""><br class=""><blockquote type="cite" class=""><br class="">Thanks,<br class="">Kevin<br class=""><br class=""><br class="">-- <br class="">-------------------------------------------------------------------<br class="">Kevin Swab UNIX Systems Administrator<br class="">ACNS Colorado State University<br class="">Phone: (970)491-6572 Email: <a href="mailto:Kevin.Swab@ColoState.EDU" class="">Kevin.Swab@ColoState.EDU</a><br class="">GPG Fingerprint: 7026 3F66 A970 67BD 6F17 8EB8 8A7D 142F 2392 791C<br class="">_______________________________________________<br class="">OmniOS-discuss mailing list<br class=""><a href="mailto:OmniOS-discuss@lists.omniti.com" class="">OmniOS-discuss@lists.omniti.com</a><br class="">http://lists.omniti.com/mailman/listinfo/omnios-discuss<br class=""></blockquote></blockquote><br class=""></blockquote></blockquote><br class="">-- <br class="">-------------------------------------------------------------------<br class="">Kevin Swab UNIX Systems Administrator<br class="">ACNS Colorado State University<br class="">Phone: (970)491-6572 Email: <a href="mailto:Kevin.Swab@ColoState.EDU" class="">Kevin.Swab@ColoState.EDU</a><br class="">GPG Fingerprint: 7026 3F66 A970 67BD 6F17 8EB8 8A7D 142F 2392 791C<br class=""></div></blockquote></div><br class=""><div apple-content-edited="true" class="">
<div style="color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; " class=""><span class="Apple-style-span" style="border-collapse: separate; border-spacing: 0px;"><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; " class=""><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; border-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; " class=""><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; border-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; " class=""><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; border-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; " class="">--</div><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; " class=""><br class=""></div><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; " class=""><a href="mailto:Richard.Elling@RichardElling.com" class="">Richard.Elling@RichardElling.com</a><br class="">+1-760-896-4422<br class=""><br class=""><br class=""></div></span></div></span></div></span></div></span></div>
</div>
<br class=""></body></html>