[OmniOS-discuss] [smartos-discuss] SuperStorage Server 6047R-E1R36L SAS Question

[Keith Wesolowski]

On Mon, Nov 11, 2013 at 10:09:48PM +0100, Tobias Oetiker wrote:

> they would build the system based on
> http://www.supermicro.com/products/chassis/4U/847/SC847A-R1400LP.cfm
> then. With the X9DRD-7LN4F-JBOD motherboard and some extra LSI
> controller cards.

Ok.

> > The only reason to want to bypass the expanders is if you are using SATA
> > devices.  We don't use them in this system, and do not recommend them.
> 
> we use one sata ssd for l2arc and two for zil ... you seem to get around
> the zil part by using zeus (expensive, is it worth it?). and no
> l2arc (reason?).

The ZeusRAM costs something like $2000.  By the time you've purchased a
SATA slog and 5 extra HBAs, you can't be far from that.  And the ZeusRAM
was designed and developed specifically for this function -- not true of
99.9% or more of the SATA SSDs on the market.  It has all the desired
attributes and is blazingly fast.  Perhaps most importantly, I don't
worry too much about the firmware in the thing because STEC has a good
track record with us.  I think we've had only 2 firmware bugs in 7 years
or so.  Put all that together, and yeah I'd say it's well worth it.
There's no way I'd use a SATA slog and a huge mess of DA HBAs instead.

We don't use the L2ARC because (a) it's not clear how to monetise that
in our business, and (b) it's unclear that there'd be much demand for
it.  Most of the people who fall out of DRAM (or assume that they are
because they don't bother to collect data) just insist on all-SSD pools.
A solution that is intermediate in cost and working set size seems at
this time to enjoy limited demand.

Obviously, build whatever's best for you.  But if you do go the mega-DA
route, I'll offer you at least one extra piece of advice.  You need to
implement a topo map for that system if you want the ability to have
indicators.  With that number of disks, you surely do.  That also means
you need (a) consistent SMBIOS identity for your system and (b)
consistent cabling of the 36 cables to the 5 HBAs.  If you don't have
all of this, you'll light the wrong LEDs when a fault is diagnosed;
operational hilarity is sure to ensue!