[OmniOS-discuss] Hung ZFS Pool
Brian Hechinger
wonko at 4amlunch.net
Thu Dec 17 19:05:21 UTC 2015
Ok, let’s add to the weirdness.
I destroyed the degraded pool.
I re-created it.
I then re-ran iozone.
It completed with zero errors on the pool. iozone did have some issues at the end, but the FS seems ok:
pool: zoom
state: ONLINE
scan: none requested
config:
NAME STATE READ WRITE CKSUM
zoom ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
c4t1d0s1 ONLINE 0 0 0
c5t1d0s1 ONLINE 0 0 0
errors: No known data errors
Iozone: Performance Test of File I/O
Version $Revision: 3.434 $
Compiled for 64 bit mode.
Build: Solaris10gcc-64
Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
Al Slater, Scott Rhine, Mike Wisner, Ken Goss
Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner,
Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone,
Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root,
Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer,
Vangel Bojaxhi, Ben England, Vikentsi Lapa,
Alexey Skidanov.
Run began: Thu Dec 17 13:21:59 2015
Multi_buffer. Work area 16777216 bytes
OPS Mode. Output is in operations per second.
Record Size 8 kB
SYNC Mode.
File size set to 2097152 kB
Command line used: /usr/local/bin/iozone -m -t 16 -T -O -r 8k -o -s 2G
Time Resolution = 0.000001 seconds.
Processor cache size set to 1024 kBytes.
Processor cache line size set to 32 bytes.
File stride size set to 17 * record size.
Throughput test with 16 threads
Each thread writes a 2097152 kByte file in 8 kByte records
Children see throughput for 16 initial writers = 29558.13 ops/sec
Parent sees throughput for 16 initial writers = 29467.57 ops/sec
Min throughput per thread = 1845.28 ops/sec
Max throughput per thread = 1853.28 ops/sec
Avg throughput per thread = 1847.38 ops/sec
Min xfer = 261012.00 ops
Children see throughput for 16 rewriters = 26802.94 ops/sec
Parent sees throughput for 16 rewriters = 26801.51 ops/sec
Min throughput per thread = 1671.70 ops/sec
Max throughput per thread = 1679.40 ops/sec
Avg throughput per thread = 1675.18 ops/sec
Min xfer = 260942.00 ops
Children see throughput for 16 readers = 305525.26 ops/sec
Parent sees throughput for 16 readers = 304910.58 ops/sec
Min throughput per thread = 16371.37 ops/sec
Max throughput per thread = 20084.48 ops/sec
Avg throughput per thread = 19095.33 ops/sec
Min xfer = 213905.00 ops
Children see throughput for 16 re-readers = 301510.86 ops/sec
Parent sees throughput for 16 re-readers = 301021.85 ops/sec
Min throughput per thread = 16066.28 ops/sec
Max throughput per thread = 19850.40 ops/sec
Avg throughput per thread = 18844.43 ops/sec
Min xfer = 212289.00 ops
Children see throughput for 16 reverse readers = 520691.82 ops/sec
Parent sees throughput for 16 reverse readers = 520026.68 ops/sec
Min throughput per thread = 30897.40 ops/sec
Max throughput per thread = 33412.20 ops/sec
Avg throughput per thread = 32543.24 ops/sec
Min xfer = 242448.00 ops
Children see throughput for 16 stride readers = 27067.77 ops/sec
Parent sees throughput for 16 stride readers = 27064.74 ops/sec
Min throughput per thread = 1549.09 ops/sec
Max throughput per thread = 3205.10 ops/sec
Avg throughput per thread = 1691.74 ops/sec
Min xfer = 126699.00 ops
Children see throughput for 16 random readers = 215258.98 ops/sec
Parent sees throughput for 16 random readers = 214461.71 ops/sec
Min throughput per thread = 2759.80 ops/sec
Max throughput per thread = 169551.89 ops/sec
Avg throughput per thread = 13453.69 ops/sec
Min xfer = 4281.00 ops
Children see throughput for 16 mixed workload = 8673.89 ops/sec
Parent sees throughput for 16 mixed workload = 6341.03 ops/sec
Min throughput per thread = 442.73 ops/sec
Max throughput per thread = 641.36 ops/sec
Avg throughput per thread = 542.12 ops/sec
Min xfer = 180991.00 ops
Children see throughput for 16 random writers = 4008.54 ops/sec
Parent sees throughput for 16 random writers = 3972.48 ops/sec
Min throughput per thread = 248.54 ops/sec
Max throughput per thread = 252.76 ops/sec
Avg throughput per thread = 250.53 ops/sec
Min xfer = 257769.00 ops
Children see throughput for 16 fwriters = 70222.20 ops/sec
Parent sees throughput for 16 fwriters = 65632.32 ops/sec
Min throughput per thread = 4132.12 ops/sec
Max throughput per thread = 4686.85 ops/sec
Avg throughput per thread = 4388.89 ops/sec
Min xfer = 262144.00 ops
Error in file: Found ?0? Expecting ?7979797979797979? addr 29f6770
Error in file: Found ?0? Expecting ?7979797979797979? addr 29f6770
Error in file: Position 0
Error in file: Position 0
Record # 0 Record size 8 kb
Record # 0 Record size 8 kb
where 29f6770x loop 0
where 29f6770x loop 0
I can delete and create files just fine.
Grrrr.
-brian
> On Dec 9, 2015, at 11:27 AM, Brian Hechinger <wonko at 4amlunch.net> wrote:
>
>
>> On Dec 9, 2015, at 11:22 AM, Dan McDonald <danmcd at omniti.com> wrote:
>>
>>
>>> On Dec 9, 2015, at 11:18 AM, Brian Hechinger <wonko at 4amlunch.net> wrote:
>>>
>>> It’s brand new!!
>>
>> Sometimes you get flaky HW that's new. I've had to return new spinning-rust disks, for example.
>
> Bah. :(
>
>>
>>> Also, I would expect the other slice to be affected as well? It’s been humming along just fine as SLOG with no errors:
>>>
>>> logs
>>> mirror-3 ONLINE 0 0 0
>>> c4t1d0s0 ONLINE 0 0 0
>>> c5t1d0s0 ONLINE 0 0 0
>>
>> Could just be bad luck your slog hasn't encountered the bad portion of this drive.
>
> I suppose. You think there is a maybe a good way to test this device before I try to get it RMA-ed?
>
>> Also, what OmniOS revision are you running? If you're not up to the latest November r151014 update, you may be missing some NVMe fixes.
>
> Oh right, totally forgot to do that for you:
>
> wonko at basket1:/var/adm$ head /etc/release ; uname -a
> OmniOS v11 r151016
> Copyright 2015 OmniTI Computer Consulting, Inc. All rights reserved.
> Use is subject to license terms.
> SunOS basket1 5.11 omnios-073d8c0 i86pc i386 i86pc
>
More information about the OmniOS-discuss
mailing list