[OmniOS-discuss] NFSv4 client soft lockups

[Derek Yarnell]

Hi,

I am wondering if anyone else has experience running NFSv4 from OmniOS
to RHEL (6.5) clients.  We are running it with sec=krb5p and are getting
these errors (soft lockups) at the end of this mail across all the nodes
causing them to be unresponsive after.  The OmniOS server doesn't report
anything wrong in the logs.  I have submitted a ticket to Red Hat about
this already but since it happens on all my RHEL clients at once and
references the same OmniOS server (172.19.0.98) then it may have
something to do either Omni NFS server or the interoperability of the
RHEL NFSv4 client and the Omni NFSv4 server.

Thanks,
derek

Sep 11 23:00:54 nauseated kernel: BUG: soft lockup - CPU#5 stuck for
67s! [172.19.0.98-man:12273]
Sep 11 23:00:54 nauseated kernel: Modules linked in: aesni_intel
ablk_helper cryptd lrw gf128mul glue_helper aes_x86_64 aes_generic cbc
cts nfs lockd fscache nfs_acl autofs4 rpcsec_gss_krb5 auth_rpcgss sunrpc
ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables
ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack
ip6table_filter ip6_tables ipv6 power_meter microcode iTCO_wdt
iTCO_vendor_support dcdbas serio_raw lpc_ich mfd_core sg i7core_edac
edac_core bnx2 ext4 jbd2 mbcache sr_mod cdrom sd_mod crc_t10dif
pata_acpi ata_generic ata_piix mptsas mptscsih mptbase
scsi_transport_sas dm_mirror dm_region_hash dm_log dm_mod [last
unloaded: speedstep_lib]
Sep 11 23:00:54 nauseated kernel: CPU 5
Sep 11 23:00:54 nauseated kernel: Modules linked in: aesni_intel
ablk_helper cryptd lrw gf128mul glue_helper aes_x86_64 aes_generic cbc
cts nfs lockd fscache nfs_acl autofs4 rpcsec_gss_krb5 auth_rpcgss sunrpc
ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables
ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack
ip6table_filter ip6_tables ipv6 power_meter microcode iTCO_wdt
iTCO_vendor_support dcdbas serio_raw lpc_ich mfd_core sg i7core_edac
edac_core bnx2 ext4 jbd2 mbcache sr_mod cdrom sd_mod crc_t10dif
pata_acpi ata_generic ata_piix mptsas mptscsih mptbase
scsi_transport_sas dm_mirror dm_region_hash dm_log dm_mod [last
unloaded: speedstep_lib]
Sep 11 23:00:54 nauseated kernel:
Sep 11 23:00:54 nauseated kernel: Pid: 12273, comm: 172.19.0.98-man Not
tainted 2.6.32-431.29.2.el6.x86_64 #1 Dell Inc. PowerEdge R610/0F0XJ6
Sep 11 23:00:54 nauseated kernel: RIP: 0010:[<ffffffffa02f568d>]
[<ffffffffa02f568d>] nfs4_run_state_manager+0x38d/0x620 [nfs]
Sep 11 23:00:54 nauseated kernel: RSP: 0018:ffff88042b249e90  EFLAGS:
00000206
Sep 11 23:00:54 nauseated kernel: RAX: ffff88042b249fd8 RBX:
ffff88042b249ee0 RCX: 0000000000000034
Sep 11 23:00:54 nauseated kernel: RDX: 000000000000115b RSI:
ffff880429e7ed90 RDI: ffffffffa0247568
Sep 11 23:00:54 nauseated kernel: RBP: ffffffff8100bb8e R08:
f000000000000000 R09: fbb8191d81fb3e00
Sep 11 23:00:54 nauseated kernel: R10: 0000000000000001 R11:
0000000000000000 R12: ffff88042b249ee0
Sep 11 23:00:54 nauseated kernel: R13: ffffffff8100bb8e R14:
ffffffffffffff10 R15: ffffffffa0247568
Sep 11 23:00:54 nauseated kernel: FS:  0000000000000000(0000)
GS:ffff880028240000(0000) knlGS:0000000000000000
Sep 11 23:00:54 nauseated kernel: CS:  0010 DS: 0018 ES: 0018 CR0:
000000008005003b
Sep 11 23:00:54 nauseated kernel: CR2: 00000000006e19b8 CR3:
000000042b026000 CR4: 00000000000007e0
Sep 11 23:00:54 nauseated kernel: DR0: 0000000000000000 DR1:
0000000000000000 DR2: 0000000000000000
Sep 11 23:00:54 nauseated kernel: DR3: 0000000000000000 DR6:
00000000ffff0ff0 DR7: 0000000000000400
Sep 11 23:00:54 nauseated kernel: Process 172.19.0.98-man (pid: 12273,
threadinfo ffff88042b248000, task ffff8804299b9540)
Sep 11 23:00:54 nauseated kernel: Stack:
Sep 11 23:00:54 nauseated kernel: 0000000000000286 00000000000003fb
ffff880429e7ecf9 ffff880429e7ed00
Sep 11 23:00:54 nauseated kernel: <d> ffff88042b249ee0 ffff88042a5e1898
ffff88042b249ef8 ffffffffa02f5300
Sep 11 23:00:54 nauseated kernel: <d> ffff880429e7ec00 ffff88042cc89500
ffff88042b249f40 ffffffff8109abf6
Sep 11 23:00:54 nauseated kernel: Call Trace:
Sep 11 23:00:54 nauseated kernel: [<ffffffffa02f5300>] ?
nfs4_run_state_manager+0x0/0x620 [nfs]
Sep 11 23:00:54 nauseated kernel: [<ffffffff8109abf6>] ? kthread+0x96/0xa0
Sep 11 23:00:54 nauseated kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Sep 11 23:00:54 nauseated kernel: [<ffffffff8109ab60>] ? kthread+0x0/0xa0
Sep 11 23:00:54 nauseated kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Sep 11 23:00:54 nauseated kernel: Code: 89 df e8 f7 8b 00 00 e9 e3 fc ff
ff 66 90 48 89 df e8 d8 e4 ff ff 48 83 bb f8 00 00 00 00 0f 84 f7 fc ff
ff f0 41 0f ba 2c 24 00 <19> c0 85 c0 0f 84 df fc ff ff e9 e1 fc ff ff
0f 1f 40 00 41 81
Sep 11 23:00:54 nauseated kernel: Call Trace:
Sep 11 23:00:54 nauseated kernel: [<ffffffffa02f5300>] ?
nfs4_run_state_manager+0x0/0x620 [nfs]
Sep 11 23:00:54 nauseated kernel: [<ffffffff8109abf6>] ? kthread+0x96/0xa0
Sep 11 23:00:54 nauseated kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
Sep 11 23:00:54 nauseated kernel: [<ffffffff8109ab60>] ? kthread+0x0/0xa0
Sep 11 23:00:54 nauseated kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20

-- 
Derek T. Yarnell
University of Maryland
Institute for Advanced Computer Studies