[OmniOS-discuss] How bad are these controller / io errors??

[Narayan Desai]

responses inline


On Fri, Aug 16, 2013 at 9:41 AM, <steve at linuxsuite.org> wrote:

> > We're seeing something similar on the same gear (LSI/supermicro
> expanders,
> > lsi controllers, sata drives).
> >
> > We've tried standard hardware debugging (cable reseat/replacement, etc)
> > and
> > the problem in our case seems to follow the sas expander backplane.
> >
> > We did a disk by disk migration into a different expander and they
> > stopped.
> >
> > How high are your error counts? (in our case, we were getting about
> > 1500/day/device). Is your performance impacted? (it was in our case)
> >  -nld
> >
>
>         Different expander? but still SATA behind SAS expander? On
> Supermicro 847 chassis?
>

Same model of expander, still SATA behind a SAS expander. Still supermicro
847.

        Is your setup stable, ie. works and drives don't drop out as failed?
> Performance isn't an issue here, but stability is..
>

We're having extreme performance issues, but no stability problems. The
system just does I/O slowly. (less than 50MB/s, when we should be getting
1-2 GB/s for scrubs, etc)

Moving the drives to another expander resolved the issue, though we had a
second expander start experiencing the same issues at a lower rate, so we
have more drive moves to do before we can expect to resolve the lower rate
errors.


>          It is definately a SATA behind SAS expanders issue. I did lots of
> testing
> with pools built on SAS drives they have no errors.  I also did a lot of
> stress testing
> with 20T SATA pools, they were completely unusable, scrub would
> always wipe out the pool because drives would "drop out" as failed,
> but a hardware power cycle of the  SuperMicro chassis
> would bring them all back. Then I turned off NCQ on the  LSI controller
> and everything worked fine. Couldn't get anything to fail no matter
> how hard I beat on it.
>

We haven't had any issues like this at all. We've got at least 8 of these
kinds of systems, with similar configs (SATA drives, SAS expanders,
generally not chaining expanders). Each of them has at least 80 spindles,
and generally they work reliably and perform well. (Excluding cases where
there is a bad drive in a stripe, etc) Getting upwards of 2GB/s in
aggregate off of one of these isn't hard.


>          I will start to track error rates, we are not moving much data
> yet..
>
>          Would SATA port multipliers be a better solution? Does
> Solaris/OmniOS
> support such a hardware config.
>
>            Just came across this
>
>              http://www.45drives.com/
>
>          Which I think is  a SATA port multiplier solution....
> Centos/NetBSD?? can
> it work with OMniOS?
>

I'd assume it would work as long as there is a driver for those sata
controllers, but you know what they say about assuming. These seem a bit
expensive, particularly considering they don't include any drives.
 -nld
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://omniosce.org/ml-archive/attachments/20130816/8c4e7721/attachment-0001.html>