[OmniOS-discuss] Infiniband In Omni

[Chris Zembower]

I'm actually lab'ing with this right now in OmniOS! Although using SRP, and
not to ESX initiators...

Currently I have a mini-ITX board with a single PCIe 3.0 x16 slot,
populated with a ConnectX-2. zpool is 4x1TB SSD's (on mainboard SATA 6gb/s
SATA ports) in a zvol, exported via Comstar to Windows7 SRP initiator
(OFED).

Performance is about 1.2GB/s on sequential I/O. I threw log data at it all
weekend to see if it would hiccup, so far so good. Not bad, and couldn't
have been easier to set up. Essentially plug and play, took my 3 hours from
unboxing the board to working SRP target. My currently production
environment is Linux/SCST/ConnectX-3. Performance is fine (aggregate
bandwidth from a single server is in the 10GB/s area), but configuration is
much more involved than Comstar and I'm running into some latency issues. I
wanted to give Illumos an IB test drive and see how it held up.

Fortunately, you don't need OpenSM because that switch has an integrated
subnet manager.

Not sure if those cards you're using are compatible, but if not,
ConnectX-2's are cheap and perform quite well.

Other comments are accurate, the Hermon/Tevor drivers don't seem to have
been updated since forever ago (I have yet to investigate Fujita's port of
OFED), so you can't use the newer HCA's, but everything does seem stable
(and that's what's important, right?) Even on ConnectX-2, you're still
using a fabric with twice the bandwidth of 16gb fibre channel...

Good luck, interested to hear what you learn.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://omniosce.org/ml-archive/attachments/20140929/bc835590/attachment.html>