[OmniOS-discuss] Panic under heavy I/O conditions
Richard Jahnel
Richard.Jahnel at RealPage.com
Wed Nov 22 15:20:04 UTC 2017
I have briefly touched on this panic condition before, however since upgrading to R151022ce the previous mitigation of restricting the pool to 3 specific flags is no longer working. So here I am for some further discussion and hopefully more clues with which to chase the core issue down and kill it dead.
Initial condition is to have a pool under very heavy write I/O. In our case here it is performing an eager zero during the creation of a VMDK file by an esxi fibre client.
Then while the eager zero is in progress have a scheduled snapshot destroy and creation script run. At this point there is about a 20ish percent chance you will generate a panic during the execution of that script.
All the technical readouts I have been able to generate thus far follow.
> ::status
debugging crash dump vmcore.2 (64-bit) from DALSTOR1
operating system: 5.11 omnios-r151022-5e982daae6 (i86pc)
image uuid: febd682c-2ba5-69ea-b5c4-c8a30c88ffc4
panic message: hati_pte_map: flags & HAT_LOAD_REMAP
dump content: kernel pages only
> $C
ffffd000f5d6f7b0 vpanic()
ffffd000f5d6f850 hati_pte_map+0x3ab(ffffd0341195c518, 46, ffffd000af162cd8, 8000001756ea8007, 0,
0)
ffffd000f5d6f8e0 hati_load_common+0x139(ffffd0321f9a8908, 8046000, ffffd000af162cd8, 40b, 0, 0,
1756ea8)
ffffd000f5d6f960 hat_memload+0x75(ffffd0321f9a8908, 8046000, ffffd000af162cd8, b, 0)
ffffd000f5d6fa80 segvn_faultpage+0x730(ffffd0321f9a8908, ffffd03252f19df0, 8046000,
fffffffffffff000, 0, ffffd000f5d6fb50, ffffd03400000000, ffffd03100000001, ffffd00000000002,
ffffffff00000001)
ffffd000f5d6fc50 segvn_fault+0x8e6(ffffd0321f9a8908, ffffd03252f19df0, 8046000, 1000, 1, 2)
ffffd000f5d6fd60 as_fault+0x312(ffffd0321f9a8908, ffffd0321fa22e20, 8046efc, 1, 1, 2)
ffffd000f5d6fdf0 pagefault+0x96(8046efc, 1, 2, 0)
ffffd000f5d6ff00 trap+0x30c(ffffd000f5d6ff10, 8046efc, b)
ffffd000f5d6ff10 0xfffffffffb8001d6()
> ::stack
vpanic()
hati_pte_map+0x3ab(ffffd0341195c518, 46, ffffd000af162cd8, 8000001756ea8007, 0, 0)
hati_load_common+0x139(ffffd0321f9a8908, 8046000, ffffd000af162cd8, 40b, 0, 0)
hat_memload+0x75(ffffd0321f9a8908, 8046000, ffffd000af162cd8, b, 0)
segvn_faultpage+0x730(ffffd0321f9a8908, ffffd03252f19df0, 8046000, fffffffffffff000, 0,
ffffd000f5d6fb50)
segvn_fault+0x8e6(ffffd0321f9a8908, ffffd03252f19df0, 8046000, 1000, 1, 2)
as_fault+0x312(ffffd0321f9a8908, ffffd0321fa22e20, 8046efc, 1, 1, 2)
pagefault+0x96(8046efc, 1, 2, 0)
trap+0x30c(ffffd000f5d6ff10, 8046efc, b)
0xfffffffffb8001d6()
> ::cpuinfo
ID ADDR FLG NRUN BSPL PRI RNRN KRNRN SWITCH THREAD PROC
0 fffffffffbc397a0 1f 0 0 39 no no t-0 ffffd031ff4cb060 DALV4_snapcycle.
1 ffffd031ea372b00 1f 2 0 -1 no no t-0 ffffd000f45fdc40 (idle)
2 ffffd031ea36f500 1f 0 0 -1 no no t-0 ffffd000f46f7c40 (idle)
3 ffffd031ea369540 1f 0 0 -1 no no t-0 ffffd000f47a9c40 (idle)
4 ffffd031ea368040 1f 0 0 -1 no no t-0 ffffd000f467bc40 (idle)
5 ffffd031ea4bdb00 1f 0 0 -1 no no t-0 ffffd000f4868c40 (idle)
6 ffffd031ea4b3a80 1f 1 0 -1 no no t-0 ffffd000f49afc40 (idle)
7 ffffd031ea4aa540 1f 2 0 -1 no no t-0 ffffd000f50a7c40 (idle)
8 ffffd031ea4a9040 1f 2 0 -1 no no t-0 ffffd000f51a1c40 (idle)
9 ffffd031ea60cb00 1f 2 0 -1 no no t-0 ffffd000f5253c40 (idle)
10 ffffd031ea602a80 1f 1 0 -1 no no t-0 ffffd000f5305c40 (idle)
11 fffffffffbc440a0 1b 0 0 29 no no t-0 ffffd03220006b60 DALV4_snapcycle.
> ffffd031ff4cb060::findstack -v
stack pointer for thread ffffd031ff4cb060: ffffd000f78fd720
ffffd000f78fd910 page_trylock+1()
ffffd000f78fd9c0 zfs_getpage+0x185(ffffd031ffb33e40, e2000, 1000, ffffd000f78fdbec,
ffffd000f78fdb50, 8000, ffffd0347314fbf8, fef02000, 3, ffffd0327ee27c80, 0)
ffffd000f78fda80 fop_getpage+0x7e(ffffd031ffb33e40, e2000, 1000, ffffd000f78fdbec,
ffffd000f78fdb50, 8000, ffffd0347314fbf8, fef02000, ffffd00000000003, ffffd0327ee27c80, 0)
ffffd000f78fdc50 segvn_fault+0xdfa(ffffd0321f9a8598, ffffd0347314fbf8, fef02000, 1000, 0, 3)
ffffd000f78fdd60 as_fault+0x312(ffffd0321f9a8598, ffffd0321fa229c0, fef0276b, 1, 0, 3)
ffffd000f78fddf0 pagefault+0x96(fef0276b, 0, 3, 0)
ffffd000f78fdf00 trap+0x30c(ffffd000f78fdf10, fef0276b, 0)
ffffd000f78fdf10 0xfffffffffb8001d6()
> ffffd03220006b60::findstack -v
stack pointer for thread ffffd03220006b60: ffffd000f5d6f740
ffffd000f5d6f7b0 param_preset()
ffffd000f5d6f850 hati_pte_map+0x3ab(ffffd0341195c518, 46, ffffd000af162cd8, 8000001756ea8007, 0
, 0)
ffffd000f5d6f8e0 hati_load_common+0x139(ffffd0321f9a8908, 8046000, ffffd000af162cd8, 40b, 0, 0
, 1756ea8)
ffffd000f5d6f960 hat_memload+0x75(ffffd0321f9a8908, 8046000, ffffd000af162cd8, b, 0)
ffffd000f5d6fa80 segvn_faultpage+0x730(ffffd0321f9a8908, ffffd03252f19df0, 8046000,
fffffffffffff000, 0, ffffd000f5d6fb50, ffffd03400000000, ffffd03100000001, ffffd00000000002,
ffffffff00000001)
ffffd000f5d6fc50 segvn_fault+0x8e6(ffffd0321f9a8908, ffffd03252f19df0, 8046000, 1000, 1, 2)
ffffd000f5d6fd60 as_fault+0x312(ffffd0321f9a8908, ffffd0321fa22e20, 8046efc, 1, 1, 2)
ffffd000f5d6fdf0 pagefault+0x96(8046efc, 1, 2, 0)
ffffd000f5d6ff00 trap+0x30c(ffffd000f5d6ff10, 8046efc, b)
ffffd000f5d6ff10 0xfffffffffb8001d6()
> hati_pte_map+0x3ab::dis
hati_pte_map+0x37e: movq %r12,%rdx
hati_pte_map+0x381: movq $0xfffffffffb93ae08,%rdi
hati_pte_map+0x388: xorl %eax,%eax
hati_pte_map+0x38a: call +0x665e1 <panic>
hati_pte_map+0x38f: movq $0xfffffffffb93acf8,%rdi
hati_pte_map+0x396: xorl %eax,%eax
hati_pte_map+0x398: call +0x665d3 <panic>
hati_pte_map+0x39d: movq $0xfffffffffb93ad18,%rdi
hati_pte_map+0x3a4: xorl %eax,%eax
hati_pte_map+0x3a6: call +0x665c5 <panic>
hati_pte_map+0x3ab: movq $0xfffffffffb957c1f,%rdi
hati_pte_map+0x3b2: xorl %eax,%eax
hati_pte_map+0x3b4: call +0x665b7 <panic>
hati_pte_map+0x3b9: movq $0xfffffffffb93adb0,%rdi
hati_pte_map+0x3c0: xorl %eax,%eax
hati_pte_map+0x3c2: call +0x665a9 <panic>
hati_pte_map+0x3c7: movq $0xfffffffffb93ad70,%rdi
hati_pte_map+0x3ce: xorl %eax,%eax
hati_pte_map+0x3d0: call +0x6659b <panic>
hati_pte_map+0x3d5: movq $0xfffffffffb93ad40,%rdi
hati_pte_map+0x3dc: xorl %eax,%eax
> fop_getpage+0x7e::dis
fop_getpage+0x52: movq 0x30(%rbp),%rdi
fop_getpage+0x56: movq 0x40(%rbx),%r10
fop_getpage+0x5a: movl %eax,0x10(%rsp)
fop_getpage+0x5e: movq 0x18(%rbp),%rax
fop_getpage+0x62: movq %rdi,0x20(%rsp)
fop_getpage+0x67: movq %rbx,%rdi
fop_getpage+0x6a: movq %rax,0x8(%rsp)
fop_getpage+0x6f: movq 0x10(%rbp),%rax
fop_getpage+0x73: movq %rax,(%rsp)
fop_getpage+0x77: call *0xf0(%r10)
fop_getpage+0x7e: movq 0x18(%rbx),%r12
fop_getpage+0x82: testq %r12,%r12
fop_getpage+0x85: je +0x51 <fop_getpage+0xd8>
fop_getpage+0x87: movq 0xa0(%r12),%r13
fop_getpage+0x8f: testq %r13,%r13
fop_getpage+0x92: je +0x44 <fop_getpage+0xd8>
fop_getpage+0x94: testb $0x20,0x21(%r12)
fop_getpage+0x9a: je +0x3c <fop_getpage+0xd8>
fop_getpage+0x9c: cmpl $0xb,0x28(%rbx)
fop_getpage+0xa0: je +0x36 <fop_getpage+0xd8>
fop_getpage+0xa2: leaq 0x640(%r13),%rdx
> zfs_getpage+0x185::dis
zfs_getpage+0x169: movq -0x68(%rbp),%r13
zfs_getpage+0x16d: jmp +0x9 <zfs_getpage+0x178>
zfs_getpage+0x16f: nop
zfs_getpage+0x170: subq %rbx,%r13
zfs_getpage+0x173: je +0x1c <zfs_getpage+0x191>
zfs_getpage+0x175: addq %rbx,%r12
zfs_getpage+0x178: xorl %edx,%edx
zfs_getpage+0x17a: movq %r12,%rsi
zfs_getpage+0x17d: movq %r15,%rdi
zfs_getpage+0x180: call +0x3e0abdb <page_lookup_nowait>
zfs_getpage+0x185: movq %rax,(%r14)
zfs_getpage+0x188: addq $0x8,%r14
zfs_getpage+0x18c: testq %rax,%rax
zfs_getpage+0x18f: jne -0x21 <zfs_getpage+0x170>
zfs_getpage+0x191: movq -0x70(%rbp),%rax
zfs_getpage+0x195: movl 0xc4(%rax),%ecx
zfs_getpage+0x19b: testl %ecx,%ecx
zfs_getpage+0x19d: je +0x11 <zfs_getpage+0x1b0>
zfs_getpage+0x19f: movq -0x70(%rbp),%rdx
zfs_getpage+0x1a3: movq (%rdx),%rax
zfs_getpage+0x1a6: testb $0x1,0x20(%rax)
> hati_load_common+0x139::dis
hati_load_common+0x113: call -0x8d8 <hati_mkpte>
hati_load_common+0x118: cmpq %r13,+0x419f31(%rip) <kas+0x10>
hati_load_common+0x11f: je +0x4f <hati_load_common+0x170>
hati_load_common+0x121: movq -0x70(%rbp),%rdx
hati_load_common+0x125: movl -0x74(%rbp),%esi
hati_load_common+0x128: xorl %r9d,%r9d
hati_load_common+0x12b: movl %r12d,%r8d
hati_load_common+0x12e: movq %rax,%rcx
hati_load_common+0x131: movq %rbx,%rdi
hati_load_common+0x134: call -0x529 <hati_pte_map>
hati_load_common+0x139: movq %rbx,%rdi
hati_load_common+0x13c: movl %eax,-0x80(%rbp)
hati_load_common+0x13f: call +0x609c <htable_release>
hati_load_common+0x144: movq %gs:0x18,%rdx
hati_load_common+0x14d: decb 0x302(%rdx)
hati_load_common+0x153: movl -0x80(%rbp),%eax
hati_load_common+0x156: movq -0x58(%rbp),%rbx
hati_load_common+0x15a: movq -0x50(%rbp),%r12
hati_load_common+0x15e: movq -0x48(%rbp),%r13
hati_load_common+0x162: movq -0x40(%rbp),%r14
hati_load_common+0x166: movq -0x38(%rbp),%r15
> hat_memload+0x75::dis
hat_memload+0x51: cmpq +0x419e40(%rip),%rbx <mmu+0x68>
hat_memload+0x58: jb +0x36 <hat_memload+0x90>
hat_memload+0x5a: movl %r14d,%ecx
hat_memload+0x5d: xorl %r9d,%r9d
hat_memload+0x60: movq %r12,%rdx
hat_memload+0x63: orb $0x4,%ch
hat_memload+0x66: movq %rbx,%rsi
hat_memload+0x69: movq %r13,%rdi
hat_memload+0x6c: movq %rax,(%rsp)
hat_memload+0x70: call -0x365 <hati_load_common>
hat_memload+0x75: testl %eax,%eax
hat_memload+0x77: jne +0x2a <hat_memload+0xa3>
hat_memload+0x79: movq -0x50(%rbp),%rbx
hat_memload+0x7d: movq -0x48(%rbp),%r12
hat_memload+0x81: movq -0x40(%rbp),%r13
hat_memload+0x85: movq -0x38(%rbp),%r14
hat_memload+0x89: leave
hat_memload+0x8a: ret
hat_memload+0x8b: nopl 0x0(%rax,%rax)
hat_memload+0x90: movl %r8d,%ecx
hat_memload+0x93: movl %r14d,%edx
>
******
the DALV4_snapcycle script contains the following.
#! /bin/sh
#
# The Script to do a full snapshot and replica on $svol
#
#Get and set dayand hour for NOW and PAST
NOW=$(date +"%a%d-%H")
PAST=$(TZ=GMT+125 date +"%a%d-%H")
# set variables
svol="DAL1P2/DALV4"
#record start time
echo "$svol@$NOW Start" >> /vmt/timelog.txt
date >> /vmt/timelog.txt
# Destroy the prior $PAST snapshot
echo "Destroying $svol@$PAST"
/sbin/zfs destroy $svol@$PAST
# Create the Snapshot
echo "Creating $svol@$NOW"
/sbin/zfs snapshot $svol@$NOW
# Catch errors and send an email
if test $? -ne 0
then
echo "$svol@$NOW snapshot errored" | mailx -s "DALV4 error report" imemo at ellipseinc.com
echo "$svol@$NOW died" >> /vmt/timelog.txt
date >> /vmt/timelog.txt
exit 1
fi
#record finish time
echo "$svol@$NOW Finish" >> /vmt/timelog.txt
date >> /vmt/timelog.txt
exit 0
________________________________
This message is intended only for the use of the individual(s) or entity to which it is addressed and may contain information that is privileged, confidential, and/or proprietary to RealPage and its affiliated companies. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution, forwarding or copying of this communication is prohibited without the express permission of the sender. If you have received this communication in error, please notify the sender immediately and delete the original message.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://omniosce.org/ml-archive/attachments/20171122/f2b85eb9/attachment-0001.html>
More information about the OmniOS-discuss
mailing list