[OmniOS-discuss] hardware specs for illumos storage/virtualization server

[Paul B. Henson]

On Sat, Nov 17, 2012 at 11:56:48PM +1100, Denis Cheong wrote:
> That's a lot of issues and questions to raise in the one post!

Sorry :), it was either that or post a dozen different threads ;).

> Let me start off by saying that your intentions are very similar to mine.
> My OmniOS box is the storage backend for my household, and this includes
> MythTV as well as stackloads of photos, videos, music, etc.

Cool.

> Intel S2600CO Motherboard (16 x DIMM slots, dual LGA2011)

Interesting, although I don't like that "feature key". It sounds like it
has on-board SAS, but unless you type a magic "key" into the BIOS it
doesn't work? I hate that model. I'll have to take a look at the prices
though, the supermicro board was like $500.

> 8 x 16GB Hynix DDR ECC DIMMs (total 128GB)

Ouch, 16GB sticks? They were pretty pricy compared to the 8GB.

> 2 x LSI 9201-16i 6G SAS cards

Did you flash these with the IT firmware, or do they support JBOD out of
the box?

> Norco RPC-4224 case, 24 x hotswap 6G SAS bays

I looked at a couple Norco cases. They were OK, but the Supermicro one
just seemed better built, and

> OCZ ZX Series 850W power supply (note importantly this PSU has a 30A 5V
> rail)

came with dual redundant 960w power supplies. With the Norco cases, by
the time I added in the separately purchased power supplies, the cost
differential wasn't that much.

> 13 x 3TB WD Red drives
[...]
> With respect to the NAS drives - it's too early for me to give much
> feedback on them, but they are definitely the ones to get, for the reasons
> you gave.

My only concern is performance, given they're only 5400 RPM. I just
don't know how to qualify my I/O needs enough to tell if that'll be a
problem. How do you have them set up? The newegg reviews also have a lot
of people complaining about DOA or early death, so reliability is a
concern too. With a 3 year warranty, it's not too big a deal if a couple
go belly up, as long as it's not 3 in a row before the hot spare
resilvers 8-/. Have you run bonnie++ on your setup?

> 1. It's not going to work natively, nobody that I have been able to find
> has actually got a backend build working on Solaris and kept it up to date

Heh, I had no intention of trying to build Myth natively under illumos
:). That would just be painful.

> 2. It would have been a good option, but BrandZ has gone by the wayside
> long ago

You mean the linux compatible zone? That would have been interesting, at
least performance wise. I wonder how hard it would be to bring that back
to illumos.

> 3. KVM as I have so far tested has some major limitations that would make
> it impractical, i.e. both network bandwith and disk throughput is terrible
> to the point that it will cause problems with MythTV.

Really? I've got a quad-core sandy bridge i5 desktop set up at work
right now testing out omnios and kvm. It's only on a 100Mb port right
now, but a quick iperf from omnios native outbound:

[  4]  0.0-10.0 sec   106 MBytes  88.7 Mbits/sec
[  4]  0.0-10.0 sec  87.4 MBytes  73.3 Mbits/sec

Compared to one from a fedora vm under kvm:

[  3]  0.0-10.0 sec  91.8 MBytes  76.9 Mbits/sec
[  3]  0.0-10.0 sec  86.4 MBytes  72.4 Mbits/sec

Not a ridiculous difference. What kind of throughput are you seeing? Are
you using the e1000 nic or the virtio one? I'm using e1000 right now, I
haven't had a chance to play with the virtio. I won't be in the office
next week, but the week after I'll plug this box into a gig port and do
a bit more testing. Given joyent runs this stuff commercially, I can't
imagine it has endemic performance issues.

I'm actually planning to have my mythtv storage native on zfs, and share
via nfs from the host to the guest, so the vm disk won't really be an
issue for that.

> Please keep us up to date with how you go if you do deploy a MythTV backend
> into a KVM instance though, I'm interested to see your observations on
> network throughput, since mine was so crippled..

I probably won't get this box built or running before 2013 :), but I'll
definitely do some network testing on my omnios eval the week after next
and see what it does on a gig port.

Oh, forgot to try an iperf guest->host:

[  3]  0.0-10.0 sec   920 MBytes   772 Mbits/sec

Hmm, looks like my guest to host nfs should be pretty snappy :). I'd
expect higher though, I'll have to look at that.

Thanks for the feedback...