[OmniOS-discuss] [developer] Re: [discuss] Trouble getting Supermicro board to see all its PCI-e slots
Sašo Kiselkov
skiselkov.ml at gmail.com
Wed May 1 20:28:25 EDT 2013
Hi Aakash,
First of all, thanks for the many pointers here, I cannot begin to
describe the gratitude for your help. My responses to your suggestions
below.
On 05/01/2013 08:36 PM, Aakash Saini wrote:
>> ..."Strangely, however, prtdiag sees the slots as populated:"...
>
> I don't understand how it could be pci_boot/create_bus issues; cause pci_boot
> is functioning and allocating the buses *correctly* to relevant slots. As
> stated, Left-hand has 4::x8-PCI-e slots (2-cpu1, 2-cpu2) and right-hand has
> 2::x8-PCI-e slots (2-cpu2), which we see through prtdiag details about
> slots (1-to-6) assignments.
>
> ..which means socket 00 is working correctly, and socket 01 has some
> fault!!. surprising, very much nailed correctly relating to QPI
> interconnect issues. i guess PCH/C6xxx-chipset and motherboard specs need
> to be reviewed.
>
> prtpicl should display correctly details to resource addressing, cause
> slots are assigned.
Attached, both the short and verbose version.
>> ..."The problem appears to be that Illumos doesn't recognize any PCI
> bridges beyond those attached to CPU1.
>
> yes, maybe its a PCIe-PCI bridging issue, even i doubt. try kdb debug
> state...
> ::bp pcieb`pcieb_attach
> :c
> pcieb`pcieb_dbg_print/W 0x1 --- this should display basic
> debugged information
> :c
Compiled a debug version of the pcieb module and attached the output
from this. Strangely enough, the output stopped for considerable time
between lines 21 and 22 in the output, so between lines:
pcieb(0): pwrINITCHILD: config regs setup for pci15d9,690 at 0
and
pcieb(1): pwrpcieb_pwr_disable: disabling PM
after which lots of output proceeded quickly. Could it be that the
probing code is getting stuck on and then giving up on probing a bridge
here?
> also. try with disabled (both) MSI and workaround patch to 41210/chipset
> individually.
> pcieb`pcieb_enable_msi/W 0x1
> pcieb_disable_41210_wkarnd/W 0x1
Neither setting changed anything.
> i'm not sure if there is any resource conflict issue; mdb -k | ::interrupts
> as obviously, h/w is remote!
Here's the output of that:
# echo ::interrupts | mdb -k
CPU/Vect IRQ IPL Bus Trg Type Share APIC/INT# ISR
1/0x20 9 9 PCI Lvl Fixed 1 0x0/0x9 acpi_wrapper_isr
2/0x20 11 14 PCI Lvl Fixed 1 0x0/0xb hpet_isr
3/0x20 - 7 PCI Edg MSI 1 - pcieb_intr_handler
4/0x20 - 5 PCI Edg MSI-X 1 - mrsas_isr
7/0x20 1 5 ISA Edg Fixed 1 0x0/0x1 i8042_intr
8/0x20 12 5 ISA Edg Fixed 1 0x0/0xc i8042_intr
8/0x22 - 5 PCI Edg MSI 1 - mpt_intr
12/0x20 3 12 ISA Edg Fixed 1 0x0/0x3 asyintr
13/0x20 - 4 PCI Edg MSI 1 - pcieb_intr_handler
14/0x20 - 7 PCI Edg MSI 1 - pcieb_intr_handler
15/0x20 - 4 PCI Edg MSI 1 - pcieb_intr_handler
16/0x20 - 7 PCI Edg MSI 1 - pcieb_intr_handler
17/0x20 - 4 PCI Edg MSI 1 - pcieb_intr_handler
18/0x20 16 9 PCI Lvl Fixed 1 0x0/0x10 ehci_intr
19/0x20 23 9 PCI Lvl Fixed 1 0x0/0x17 ehci_intr
20/0x20 - 6 PCI Edg MSI-X 1 - igb_intr_tx_other
21/0x20 - 6 PCI Edg MSI-X 1 - igb_intr_rx
all/0xf0 - 15 - Edg IPI 1 - xc_serv
all/0xf1 - 11 - Edg IPI 0 - poke_cpu
all/0xf2 - 14 - Edg IPI 1 - kcpc_hw_overflow_intr
all/0xf3 - 15 - Edg IPI 1 - apic_error_intr
all/0xf4 - 2 - Edg IPI 1 - cmi_cmci_trap
all/0xf5 - 14 - Edg IPI 1 - cbe_fire
all/0xf6 - 14 - Edg IPI 1 - cbe_fire
Hope this provides more insight. If it helps, I can arrange for direct
access to the machine (both to the OS and the BMC).
Cheers,
--
Saso
-------------- next part --------------
/ (picl, f900000001)
platform (upa, f900000005)
scsi_vhci (devctl, f90000005c)
pci (pciex, f900000079)
pci15d9,720 (obp-device, f90000008e)
pci8086,3c02 (pciex, f9000000af)
pci15d9,690 (obp-device, f9000000d6)
sd (block, f900000108)
pci8086,3c04 (pciex, f900000126)
pci15d9,10fb (obp-device, f90000014e)
pci15d9,10fb (obp-device, f900000179)
pci8086,3c06 (pciex, f9000001a3)
pci10e3,8113 (pci, f9000001ca)
display (display, f9000001ee)
pci8086,3c07 (pciex, f900000214)
pci15d9,1521 (obp-device, f90000023b)
pci15d9,1521 (obp-device, f900000266)
pci8086,3c08 (pciex, f900000290)
pci103c,322d (obp-device, f9000002b8)
pci8086,3c0a (pciex, f9000002f6)
pci15d9,720 (obp-device, f90000031a)
pci15d9,720 (obp-device, f90000033c)
pci15d9,720 (obp-device, f90000035e)
pci15d9,720 (obp-device, f900000380)
pci15d9,720 (obp-device, f9000003a2)
pci15d9,720 (obp-device, f9000003c4)
pci15d9,720 (obp-device, f9000003e6)
pci15d9,720 (obp-device, f900000408)
pci15d9,720 (obp-device, f90000042a)
pci15d9,720 (obp-device, f90000044a)
pci15d9,720 (obp-device, f90000046a)
pci8086,1d3e (pciex, f900000487)
pci15d9,720 (obp-device, f9000004ad)
pci15d9,720 (obp-device, f9000004cf)
pci15d9,720 (obp-device, f9000004ee)
pci15d9,720 (obp-device, f90000050d)
hub (obp-device, f900000537)
device (obp-device, f900000563)
mouse (obp-device, f900000583)
keyboard (obp-device, f90000059b)
pci15d9,720 (obp-device, f9000005b3)
hub (obp-device, f9000005dd)
storage (obp-device, f900000609)
disk (block, f900000637)
pci8086,244e (pci, f900000657)
isa (isa, f900000679)
motherboard (obp-device, f90000069c)
i8042 (obp-device, f9000006a6)
mouse (obp-device, f9000006b1)
asy (serial, f9000006c0)
asy (serial, f9000006cf)
motherboard (obp-device, f9000006de)
motherboard (obp-device, f9000006e8)
pci15d9,720 (obp-device, f9000006f2)
pci15d9,720 (obp-device, f900000723)
pci15d9,720 (obp-device, f900000742)
pci (pciex, f900000778)
pci15d9,720 (obp-device, f90000078b)
pci15d9,720 (obp-device, f9000007a7)
pci15d9,720 (obp-device, f9000007c7)
pci15d9,720 (obp-device, f9000007e7)
pci15d9,720 (obp-device, f900000803)
pci15d9,720 (obp-device, f900000823)
pci15d9,720 (obp-device, f900000843)
pci15d9,720 (obp-device, f90000085f)
pci15d9,720 (obp-device, f90000087b)
pci15d9,720 (obp-device, f900000897)
pci15d9,720 (obp-device, f9000008b3)
pci15d9,720 (obp-device, f9000008cf)
pci15d9,720 (obp-device, f9000008eb)
pci15d9,720 (obp-device, f900000907)
pci15d9,720 (obp-device, f900000923)
pci15d9,720 (obp-device, f90000093f)
pci15d9,720 (obp-device, f90000095b)
pci15d9,720 (obp-device, f900000977)
pci15d9,720 (obp-device, f900000993)
pci15d9,720 (obp-device, f9000009af)
pci15d9,720 (obp-device, f9000009cb)
pci15d9,720 (obp-device, f9000009e7)
pci15d9,720 (obp-device, f900000a03)
pci15d9,720 (obp-device, f900000a1f)
pci15d9,720 (obp-device, f900000a3f)
pci15d9,720 (obp-device, f900000a5f)
pci15d9,720 (obp-device, f900000a7f)
pci15d9,720 (obp-device, f900000a9f)
pci15d9,720 (obp-device, f900000abf)
pci15d9,720 (obp-device, f900000adf)
pci15d9,720 (obp-device, f900000afb)
pci15d9,720 (obp-device, f900000b1b)
pci15d9,720 (obp-device, f900000b3b)
pci15d9,720 (obp-device, f900000b5b)
pci15d9,720 (obp-device, f900000b7b)
pci15d9,720 (obp-device, f900000b9b)
pci15d9,720 (obp-device, f900000bbb)
pci15d9,720 (obp-device, f900000bdb)
pci8086,0 (obp-device, f900000bfb)
pci15d9,720 (obp-device, f900000c17)
pci15d9,720 (obp-device, f900000c33)
pci15d9,720 (obp-device, f900000c4f)
pci15d9,720 (obp-device, f900000c6b)
pci15d9,720 (obp-device, f900000c87)
pci (pciex, f900000ca3)
pci8086,3c01 (pciex, f900000cb3)
pci8086,3c02 (pciex, f900000cd6)
pci8086,3c04 (pciex, f900000cf9)
pci103c,322d (obp-device, f900000d1d)
pci8086,3c06 (pciex, f900000d40)
pci8086,3c08 (pciex, f900000d64)
pci8086,3c0a (pciex, f900000d88)
pci15d9,720 (obp-device, f900000dac)
pci15d9,720 (obp-device, f900000dce)
pci15d9,720 (obp-device, f900000df0)
pci15d9,720 (obp-device, f900000e12)
pci15d9,720 (obp-device, f900000e34)
pci15d9,720 (obp-device, f900000e56)
pci15d9,720 (obp-device, f900000e78)
pci15d9,720 (obp-device, f900000e9a)
pci15d9,720 (obp-device, f900000ebc)
pci15d9,720 (obp-device, f900000edc)
pci15d9,720 (obp-device, f900000efc)
fw (acpirootnex, f900000f19)
sb (acpidevice, f900000f2f)
socket (acpicontainer, f900000f45)
cpu (cpu, f900000f5d)
cpu (cpu, f900000f9c)
cpu (cpu, f900000fdb)
cpu (cpu, f90000101a)
cpu (cpu, f900001059)
cpu (cpu, f900001098)
cpu (cpu, f9000010d7)
cpu (cpu, f900001116)
cpu (cpu, f900001155)
cpu (cpu, f900001194)
cpu (cpu, f9000011d3)
cpu (cpu, f900001212)
socket (acpicontainer, f900001251)
cpu (cpu, f900001269)
cpu (cpu, f9000012a8)
cpu (cpu, f9000012e7)
cpu (cpu, f900001326)
cpu (cpu, f900001365)
cpu (cpu, f9000013a4)
cpu (cpu, f9000013e3)
cpu (cpu, f900001422)
cpu (cpu, f900001461)
cpu (cpu, f9000014a0)
cpu (cpu, f9000014df)
cpu (cpu, f90000151e)
agpgart (agpgart, f900001580)
pseudo (devctl, f900001593)
zconsnex (devctl, f90000159c)
fct (admin, f9000015a5)
stmf (admin, f9000015ad)
stmf_sbd (admin, f9000015b5)
ppm (ppm, f9000015be)
power (power_button, f90000164e)
obp (picl, f900000059)
ioapics (ioapics, f900000760)
ioapic (ioapic, f900000766)
ioapic (ioapic, f90000076f)
used-resources (used-resources, f90000155d)
iscsi (iscsi, f900001567)
options (options, f900001589)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: prtpicl-v.txt.gz
Type: application/x-gzip
Size: 18878 bytes
Desc: not available
URL: <https://omniosce.org/ml-archive/attachments/20130502/f75eca5c/attachment-0001.bin>
-------------- next part --------------
Loading kmdb...
Welcome to kmdb
kmdb: unable to determine terminal type: assuming `vt100'
Loaded modules: [ unix krtld genunix ]
[0]> ::bp pcieb`pcieb_attach
[0]> :c
SunOS Release 5.11 Version omnios-bc85f2d 64-bit
Copyright (c) 1983, 2010, Oracle and/or its affiliates. All rights reserved.
Loaded modules: [ scsi_vhci mac uppc zfs apix specfs pcplusmp cpu.generic ]
kmdb: stop at pcieb`pcieb_attach
kmdb: target stopped at:
pcieb`pcieb_attach: pushq %rbp
[0]> pcieb`pcieb_dbg_print/W 0x1
pcieb`pcieb_dbg_print: 0 = 0x1
[0]> ::delete 1
[0]> :c
pcieb(0): pwrpcieb_pwr_disable: disabling PM
pcieb(0): attachpcieb_intr_init: Attaching MSI handler
pcieb(0): attachbdf 0xf7ad82a9: ddi_intr_get_nintrs: nintrs 8pcieb(0): attachVID:0x8086 DID:0x3c02 RID:0x7 bdf=0x8
pcieb(0): pwrINITCHILD: config regs setup for pci15d9,690 at 0
pcieb(1): pwrpcieb_pwr_disable: disabling PM
pcieb(1): attachpcieb_intr_init: Attaching MSI handler
pcieb(1): attachbdf 0xf7ad82a9: ddi_intr_get_nintrs: nintrs 16pcieb(1): attachVID:0x8086 DID:0x3c04 RID:0x7 bdf=0x10
pcieb(2): pwrpcieb_pwr_disable: disabling PM
pcieb(2): attachpcieb_intr_init: Attaching MSI handler
pcieb(2): attachbdf 0xf7ad82a9: ddi_intr_get_nintrs: nintrs 18pcieb(2): attachVID:0x8086 DID:0x3c06 RID:0x7 bdf=0x12
pcieb(3): pwrpcieb_pwr_disable: disabling PM
pcieb(3): attachpcieb_intr_init: Attaching MSI handler
pcieb(3): attachbdf 0xf7ad82a9: ddi_intr_get_nintrs: nintrs 19pcieb(3): attachVID:0x8086 DID:0x3c07 RID:0x7 bdf=0x13
pcieb(4): pwrpcieb_pwr_disable: disabling PM
pcieb(4): attachpcieb_intr_init: Attaching MSI handler
pcieb(4): attachbdf 0xf7ad82a9: ddi_intr_get_nintrs: nintrs 24pcieb(4): attachVID:0x8086 DID:0x3c08 RID:0x7 bdf=0x18
pcieb(6): pwrpcieb_pwr_disable: disabling PM
pcieb(6): attachpcieb_intr_init: Attaching MSI handler
pcieb(6): attachbdf 0xf7ad82a9: ddi_intr_get_nintrs: nintrs 136pcieb(6): attachVID:0x8086 DID:0x1d3e RID:0x6 bdf=0x88
pcieb(2): pwrINITCHILD: config regs setup for pci10e3,8113 at 0
pcieb(7): pwrpcieb_pwr_disable: disabling PM
pcieb(7): attachpcieb_intr_init: Attaching INTx handler
pcieb(7): attachVID:0x10e3 DID:0x8113 RID:0x1 bdf=0x700
pcieb(7): pwrINITCHILD: config regs setup for display at 1
pcieb(3): pwrINITCHILD: config regs setup for pci15d9,1521 at 0
Hostname: head2
pcieb(3): pwrINITCHILD: config regs setup for pci15d9,1521 at 0,1
pcieb(1): pwrINITCHILD: config regs setup for pci15d9,10fb at 0
pcieb(1): pwrINITCHILD: config regs setup for pci15d9,10fb at 0,1
pcieb(4): pwrINITCHILD: config regs setup for pci103c,322d at 0
head2 console login:
More information about the OmniOS-discuss
mailing list