[OmniOS-discuss] Trouble getting Supermicro board to see all its PCI-e slots

Tue Apr 30 06:16:57 EDT 2013

I've got a bunch of new Supermicro X9DRW-7TPF+ boards, with on-board
10-gigy sweetness by Intel. The motherboard is loaded with two Intel
E5-2630 CPUs with 128GB of RAM. The boards come in the proprietary WIO
form factor from Supermicro, so to get to the PCI-e slots, I'm using the
appropriate riser cards, to get to them. The risers are:

(looking from the front of the server)
Left-hand side:	 RSC-R2UW-4E8
                 four x8 PCI-e slots (two to CPU1, two to CPU2)
Right-hand side: RSC-R2UW-2E8R
                 two x8 PCI-e slots (two to CPU2)

However, and here's the kicker, I can't get OmniOS bloody to see
anything plugged into any slot other than those that lead to CPU1 (I'm
trying a pair of LSI SAS HBAs). And yet the BIOS sees everything just
fine and I can access the cards, the boot ROMs get installed properly
and I can see all of the attached disks. Allocation of device data above
4G is disabled in the BIOS. The "lspci" utility from
pkg:/system/pciutils doesn't see them either.

Strangely, however, prtdiag sees the slots as populated:

# prtdiag -v
...
==== Upgradeable Slots ====================================

ID  Status    Type             Description
--- --------- ---------------- ----------------------------
1   available Unknown          CPU1 Slot1
2   available Unknown          CPU1 Slot2
3   in use    Unknown          CPU2 Slot3
4   in use    Unknown          CPU2 Slot4
5   available Unknown          CPU2 Slot5
6   available Unknown          CPU2 Slot6

(the cards work in slots 1 & 2, but nowhere else)

Motherboard manual:
http://www.supermicro.com/manuals/motherboard/C606_602/MNL-1420.pdf

Can anybody help please? I'm seriously out of ideas here...

Here's the dmesg on anything PCI related:

# dmesg  | grep -i pci
Apr 30 05:12:32 head1 npe: [ID 236367 kern.info] PCI Express-device:
pci8086,3c04 at 2, pcieb1
Apr 30 05:12:32 head1 genunix: [ID 936769 kern.info] pcieb1 is
/pci at 0,0/pci8086,3c04 at 2
Apr 30 05:12:32 head1 npe: [ID 236367 kern.info] PCI Express-device:
pci8086,3c06 at 2,2, pcieb2
Apr 30 05:12:32 head1 genunix: [ID 936769 kern.info] pcieb2 is
/pci at 0,0/pci8086,3c06 at 2,2
Apr 30 05:12:32 head1 npe: [ID 236367 kern.info] PCI Express-device:
pci8086,3c07 at 2,3, pcieb3
Apr 30 05:12:32 head1 genunix: [ID 936769 kern.info] pcieb3 is
/pci at 0,0/pci8086,3c07 at 2,3
Apr 30 05:12:32 head1 npe: [ID 236367 kern.info] PCI Express-device:
pci8086,1d3e at 11, pcieb6
Apr 30 05:12:32 head1 genunix: [ID 936769 kern.info] pcieb6 is
/pci at 0,0/pci8086,1d3e at 11
Apr 30 05:12:32 head1 npe: [ID 236367 kern.info] PCI Express-device:
pci8086,244e at 1e, pci_pci0
Apr 30 05:12:32 head1 genunix: [ID 936769 kern.info] pci_pci0 is
/pci at 0,0/pci8086,244e at 1e
Apr 30 05:12:32 head1 pcieb: [ID 586369 kern.info] PCIE-device:
pci10e3,8113 at 0, pcieb7
Apr 30 05:12:32 head1 npe: [ID 236367 kern.info] PCI Express-device:
pci10e3,8113 at 0, pcieb7
Apr 30 05:12:32 head1 genunix: [ID 936769 kern.info] pcieb7 is
/pci at 0,0/pci8086,3c06 at 2,2/pci10e3,8113 at 0
Apr 30 05:12:33 head1 npe: [ID 236367 kern.info] PCI Express-device:
pci15d9,720 at 1a, ehci0
Apr 30 05:12:33 head1 genunix: [ID 936769 kern.info] ehci0 is
/pci at 0,0/pci15d9,720 at 1a
Apr 30 05:12:34 head1 npe: [ID 236367 kern.info] PCI Express-device:
pci15d9,720 at 1d, ehci1
Apr 30 05:12:34 head1 genunix: [ID 936769 kern.info] ehci1 is
/pci at 0,0/pci15d9,720 at 1d
Apr 30 05:12:34 head1 genunix: [ID 936769 kern.info] mouse80420 is
/pci at 0,0/isa at 1f/i8042 at 1,60/mouse at 1
Apr 30 05:12:37 head1 genunix: [ID 936769 kern.info] hubd0 is
/pci at 0,0/pci15d9,720 at 1a/hub at 1
Apr 30 05:12:37 head1 genunix: [ID 408114 kern.info]
/pci at 0,0/pci15d9,720 at 1a/hub at 1 (hubd0) online
Apr 30 05:12:37 head1 genunix: [ID 936769 kern.info] hubd1 is
/pci at 0,0/pci15d9,720 at 1d/hub at 1
Apr 30 05:12:37 head1 genunix: [ID 408114 kern.info]
/pci at 0,0/pci15d9,720 at 1d/hub at 1 (hubd1) online
Apr 30 05:12:38 head1 genunix: [ID 936769 kern.info] usb_mid0 is
/pci at 0,0/pci15d9,720 at 1a/hub at 1/device at 3
Apr 30 05:12:38 head1 genunix: [ID 408114 kern.info]
/pci at 0,0/pci15d9,720 at 1a/hub at 1/device at 3 (usb_mid0) online
Apr 30 05:12:38 head1 genunix: [ID 936769 kern.info] hid0 is
/pci at 0,0/pci15d9,720 at 1a/hub at 1/device at 3/mouse at 0
Apr 30 05:12:38 head1 genunix: [ID 408114 kern.info]
/pci at 0,0/pci15d9,720 at 1a/hub at 1/device at 3/mouse at 0 (hid0) online
Apr 30 05:12:38 head1 genunix: [ID 936769 kern.info] hid1 is
/pci at 0,0/pci15d9,720 at 1a/hub at 1/device at 3/keyboard at 1
Apr 30 05:12:38 head1 genunix: [ID 408114 kern.info]
/pci at 0,0/pci15d9,720 at 1a/hub at 1/device at 3/keyboard at 1 (hid1) online
Apr 30 05:12:41 head1 genunix: [ID 936769 kern.info] asy0 is
/pci at 0,0/isa at 1f/asy at 1,3f8
Apr 30 05:12:41 head1 genunix: [ID 936769 kern.info] asy1 is
/pci at 0,0/isa at 1f/asy at 1,2f8

--
Saso