[OmniOS-discuss] [smartos-discuss] SuperStorage Server 6047R-E1R36L SAS Question

[Matt Weiss]

On 11/11/2013 4:56 PM, Keith Wesolowski wrote:
> On Mon, Nov 11, 2013 at 10:09:48PM +0100, Tobias Oetiker wrote:
>
>> they would build the system based on
>> http://www.supermicro.com/products/chassis/4U/847/SC847A-R1400LP.cfm
>> then. With the X9DRD-7LN4F-JBOD motherboard and some extra LSI
>> controller cards.
> Ok.
>
>>> The only reason to want to bypass the expanders is if you are using SATA
>>> devices.  We don't use them in this system, and do not recommend them.
>> we use one sata ssd for l2arc and two for zil ... you seem to get around
>> the zil part by using zeus (expensive, is it worth it?). and no
>> l2arc (reason?).
> The ZeusRAM costs something like $2000.  By the time you've purchased a
> SATA slog and 5 extra HBAs, you can't be far from that.  And the ZeusRAM
> was designed and developed specifically for this function -- not true of
> 99.9% or more of the SATA SSDs on the market.  It has all the desired
> attributes and is blazingly fast.  Perhaps most importantly, I don't
> worry too much about the firmware in the thing because STEC has a good
> track record with us.  I think we've had only 2 firmware bugs in 7 years
> or so.  Put all that together, and yeah I'd say it's well worth it.
> There's no way I'd use a SATA slog and a huge mess of DA HBAs instead.
>
> We don't use the L2ARC because (a) it's not clear how to monetise that
> in our business, and (b) it's unclear that there'd be much demand for
> it.  Most of the people who fall out of DRAM (or assume that they are
> because they don't bother to collect data) just insist on all-SSD pools.
> A solution that is intermediate in cost and working set size seems at
> this time to enjoy limited demand.
>
> Obviously, build whatever's best for you.  But if you do go the mega-DA
> route, I'll offer you at least one extra piece of advice.  You need to
> implement a topo map for that system if you want the ability to have
> indicators.  With that number of disks, you surely do.  That also means
> you need (a) consistent SMBIOS identity for your system and (b)
> consistent cabling of the 36 cables to the 5 HBAs.  If you don't have
> all of this, you'll light the wrong LEDs when a fault is diagnosed;
> operational hilarity is sure to ensue!
> _______________________________________________
> OmniOS-discuss mailing list
> OmniOS-discuss at lists.omniti.com
> http://lists.omniti.com/mailman/listinfo/omnios-discuss

I am putting in production TODAY in fact this exact motherboard and a 
ZuesRAM drive.

I have been testing / benchmarking this system for a couple of months 
using an Intel SSD for a zil.  Bought the ZuesRAM before final 
production and benchmarked again.

The system has been stable so far, I do not have any issues with it to 
date.  I bought 1 lsi 9207-8i card for about $250 on newegg, just to 
have as a spare of the onboard happens to go out for some reason.  With 
the power of ZFS and being able to just import your pool elsewhere if 
need be, I really don't understand why you would want 5 HBA's.  It 
sounds like a load of bull to me.

The ZuesRAM, so far I am a fan, it is getting significantly better 
performance than the Intel 520 I used for testing, which of course is no 
surprise.  I also tested an Intel 335.  I had no intention on those 
being permanent, just wanted something to test with.  The Zues is very 
very stable, the IOPS and transfer rates are constant, whereas the SSD 
zil would fluctuate up and down a bit.

7x  - 2 vdev mirrors, and a zuesram with an All-In-One Napp-it Omni OS 
box, I am getting 40k IOPS with vmware IO Analyzer.