[OmniOS-discuss] omnios vs xen experiences

[PÁSZTOR György]

Hi,

Recently I started to build a new home nas, and I thought it would be a
good idea to use xen on the bare metal. On my previous nas (Microserver
Gen7 with amd's NL54 cpu -> no kvm) I used virtualbox, to run VMs.
My new motherboard is a supermicro with integrated intel cards supporting
pci sr-iov and other fancy stuff.

I looked around, and alpine linux (3.6.2) seemed a good choice for dom0 os:
small, can run from ramdisk (os can be on a pendrive), etc. and they have a
really fresh xen.

That's where the problems started. Following old google hits and my old
memories, I was able to install omnios in both pv and hvm mode into a vm.
The only thing what didn't really worked: networking.
igbevf and ixgbevf drivers seems missing from illumos.
xnf driver just crashed. First I thought it's xnf's problem, so I tried
older omnios, latest openindiana build, etc. if it worked somewhere, but it
always crashed.
Then I tried to find out which xen version brought in a change which broke
illumos's xnf driver, so I tried alpine 2.6, 3.0 -> xnf worked.
alpine 3.3 didn't even boot. (maybe didn't like supermicro's virtual cd
emulation)
alpine 3.4 with xen 4.6 -> fail
alpine 3.2 with xen 4.5.1 -> still works.

So something happened with xen's interface between 4.5.1 and 4.6 which
makes illumos's xnf driver crashing.

If I can get some mentorship from one of the omniosce developers, I'd be
glad if I can fix the problem in illumos.
Would someone help me on this?

Now plan to put some local disk into the new nas (that's the only part,
what I don't have yet :D) and setup a build environment.
I saw, that there is a complete wiki article about that in the omniti wiki.

It's not urgent for me to "upgrade" to the new nas. It's completely fine
for me if it won't go into "production" for a couple of months.

Btw.: I tried to directly assign a complete (not a vf) card, and that
worked with the ixgbe driver perfectly.

Another foggy part: https://wiki.xenproject.org/wiki/Storage_driver_domains
Here the doc write it works with FreeBSD oob.
It seems, it's also problematic with OmniOS.
I tried that with the 3.6.2 alpine which has xen 4.8.1

Regards,
Gyu