Hi,
My name is Kobi and I work with a company called Evogene.
We have a storage server based on a Supermicro X9DRD-7LN4F (Intel(R)
Xeon(R) CPU E5-2609, 64GB RAM) equipped with Intel X520-DA2 10Gb
Ethernet card.
Two weeks ago we encountered a problem which regards Omnios (r15012) and
OpenIndiana (151_a8) kernel stability.
The system is a clean install of r151012, only configuration is
networking and zpool (raidz3 1 vdev of 8 disks of data and 3 parity).
The storage has a single NFS share mounted (nfsvers=3) on an 8 grid
nodes (running CentOS6.6) with the following mount options :
async,tcp,vers=3,noacl,timeo=20,rw,bg,hard,intr,rsize=65536,wsize=65536 0 0.
Under normal conditions (random reads and writes) we get great
performance and stability but if we increase nfs usage the Kernel memory
usage increases until the machine eventually halts.
To test it we used on each node iozone ( Version $Revision: 3.394 )
which simultaneously write and reads 12 files with a size of 500MB.
(iozone -r 64k -i 0 -i 1 -s 500m -t 12 -F
/evolon2/iozone/`hostname`.{1..12}).
While the iozone's are running the memory of Kernel (as seen by echo
'::memstat'| mdb -k) is steadily increasing until the machine is dead.
For clarification, when the machine is dead/halts:
There is no crash files.
There is no new messages in /var/adm/messages.
It doesn't respond to pings (and obviously other network services).
Waiting doesn't help.
It can only be brought to life by power cycling it.
I attached 3 files: memstat, top and arc-kstat which I hope will assist
to understand why the storage died/halted
Thank you in advance,
Kobi.
-------------- next part --------------
Thu Dec 4 16:14:02 IST 2014
Page Summary Pages MB %Tot
------------ ---------------- ---------------- ----
Kernel 5667640 22139 34%
ZFS File Data 1171645 4576 7%
Anon 26071 101 0%
Exec and libs 1181 4 0%
Page cache 4609 18 0%
Free (cachelist) 7723 30 0%
Free (freelist) 9887541 38623 59%
Total 16766410 65493
Physical 16766409 65493
Thu Dec 4 16:14:06 IST 2014
Page Summary Pages MB %Tot
------------ ---------------- ---------------- ----
Kernel 6380424 24923 38%
ZFS File Data 1171645 4576 7%
Anon 26096 101 0%
Exec and libs 1175 4 0%
Page cache 4610 18 0%
Free (cachelist) 7729 30 0%
Free (freelist) 9174731 35838 55%
Total 16766410 65493
Physical 16766409 65493
Thu Dec 4 16:14:09 IST 2014
Page Summary Pages MB %Tot
------------ ---------------- ---------------- ----
Kernel 6524733 25487 39%
ZFS File Data 1184093 4625 7%
Anon 26096 101 0%
Exec and libs 1181 4 0%
Page cache 4612 18 0%
Free (cachelist) 7722 30 0%
Free (freelist) 9017973 35226 54%
Total 16766410 65493
Physical 16766409 65493
Thu Dec 4 16:14:12 IST 2014
Page Summary Pages MB %Tot
------------ ---------------- ---------------- ----
Kernel 7290171 28477 43%
ZFS File Data 1184093 4625 7%
Anon 26122 102 0%
Exec and libs 1181 4 0%
Page cache 4612 18 0%
Free (cachelist) 7723 30 0%
Free (freelist) 8252508 32236 49%
Total 16766410 65493
Physical 16766409 65493
Thu Dec 4 16:14:16 IST 2014
Page Summary Pages MB %Tot
------------ ---------------- ---------------- ----
Kernel 7456599 29127 44%
ZFS File Data 1190237 4649 7%
Anon 26122 102 0%
Exec and libs 1181 4 0%
Page cache 4613 18 0%
Free (cachelist) 7723 30 0%
Free (freelist) 8079935 31562 48%
Total 16766410 65493
Physical 16766409 65493
Thu Dec 4 16:14:19 IST 2014
Page Summary Pages MB %Tot
------------ ---------------- ---------------- ----
Kernel 7480950 29222 45%
ZFS File Data 1513949 5913 9%
Anon 26178 102 0%
Exec and libs 1180 4 0%
Page cache 4615 18 0%
Free (cachelist) 7724 30 0%
Free (freelist) 7731814 30202 46%
Total 16766410 65493
Physical 16766409 65493
Thu Dec 4 16:14:25 IST 2014
Page Summary Pages MB %Tot
------------ ---------------- ---------------- ----
Kernel 7481258 29223 45%
ZFS File Data 1513949 5913 9%
Anon 26174 102 0%
Exec and libs 1181 4 0%
Page cache 4616 18 0%
Free (cachelist) 7723 30 0%
Free (freelist) 7731509 30201 46%
Total 16766410 65493
Physical 16766409 65493
Thu Dec 4 16:14:28 IST 2014
Page Summary Pages MB %Tot
------------ ---------------- ---------------- ----
Kernel 7482405 29228 45%
ZFS File Data 1577181 6160 9%
Anon 26200 102 0%
Exec and libs 1181 4 0%
Page cache 4617 18 0%
Free (cachelist) 7723 30 0%
Free (freelist) 7667103 29949 46%
Total 16766410 65493
Physical 16766409 65493
Thu Dec 4 16:14:31 IST 2014
Page Summary Pages MB %Tot
------------ ---------------- ---------------- ----
Kernel 7482519 29228 45%
ZFS File Data 1577181 6160 9%
Anon 26226 102 0%
Exec and libs 1181 4 0%
Page cache 4618 18 0%
Free (cachelist) 7723 30 0%
Free (freelist) 7666962 29949 46%
Total 16766410 65493
Physical 16766409 65493
Thu Dec 4 16:14:35 IST 2014
Page Summary Pages MB %Tot
------------ ---------------- ---------------- ----
Kernel 7482590 29228 45%
ZFS File Data 1577181 6160 9%
Anon 26226 102 0%
Exec and libs 1180 4 0%
Page cache 4619 18 0%
Free (cachelist) 7723 30 0%
Free (freelist) 7666891 29948 46%
Total 16766410 65493
Physical 16766409 65493
Thu Dec 4 16:14:38 IST 2014
Page Summary Pages MB %Tot
------------ ---------------- ---------------- ----
Kernel 7482751 29229 45%
ZFS File Data 1577181 6160 9%
Anon 26252 102 0%
Exec and libs 1181 4 0%
Page cache 4619 18 0%
Free (cachelist) 7723 30 0%
Free (freelist) 7666703 29948 46%
Total 16766410 65493
Physical 16766409 65493
Thu Dec 4 16:14:41 IST 2014
Page Summary Pages MB %Tot
------------ ---------------- ---------------- ----
Kernel 7482755 29229 45%
ZFS File Data 1577181 6160 9%
Anon 26252 102 0%
Exec and libs 1181 4 0%
Page cache 4622 18 0%
Free (cachelist) 7723 30 0%
Free (freelist) 7666696 29948 46%
Total 16766410 65493
Physical 16766409 65493
Thu Dec 4 16:14:44 IST 2014
Page Summary Pages MB %Tot
------------ ---------------- ---------------- ----
Kernel 7483021 29230 45%
ZFS File Data 1577181 6160 9%
Anon 26278 102 0%
Exec and libs 1181 4 0%
Page cache 4623 18 0%
Free (cachelist) 7723 30 0%
Free (freelist) 7666403 29946 46%
Total 16766410 65493
Physical 16766409 65493
Thu Dec 4 16:14:48 IST 2014
Page Summary Pages MB %Tot
------------ ---------------- ---------------- ----
Kernel 7483581 29232 45%
ZFS File Data 1577181 6160 9%
Anon 26303 102 0%
Exec and libs 1181 4 0%
Page cache 4622 18 0%
Free (cachelist) 7724 30 0%
Free (freelist) 7665818 29944 46%
Total 16766410 65493
Physical 16766409 65493
Thu Dec 4 16:14:51 IST 2014
Page Summary Pages MB %Tot
------------ ---------------- ---------------- ----
Kernel 7645816 29866 46%
ZFS File Data 1577181 6160 9%
Anon 26328 102 0%
Exec and libs 1181 4 0%
Page cache 4624 18 0%
Free (cachelist) 7723 30 0%
Free (freelist) 7503557 29310 45%
Total 16766410 65493
Physical 16766409 65493
Thu Dec 4 16:14:55 IST 2014
Page Summary Pages MB %Tot
------------ ---------------- ---------------- ----
Kernel 7645904 29866 46%
ZFS File Data 1577181 6160 9%
Anon 26328 102 0%
Exec and libs 1181 4 0%
Page cache 4625 18 0%
Free (cachelist) 7723 30 0%
Free (freelist) 7503468 29310 45%
Total 16766410 65493
Physical 16766409 65493
Thu Dec 4 16:14:58 IST 2014
Page Summary Pages MB %Tot
------------ ---------------- ---------------- ----
Kernel 7753278 30286 46%
ZFS File Data 1577181 6160 9%
Anon 26355 102 0%
Exec and libs 1181 4 0%
Page cache 4626 18 0%
Free (cachelist) 7723 30 0%
Free (freelist) 7396066 28890 44%
Total 16766410 65493
Physical 16766409 65493
Thu Dec 4 16:15:01 IST 2014
Page Summary Pages MB %Tot
------------ ---------------- ---------------- ----
Kernel 8085909 31585 48%
ZFS File Data 1577181 6160 9%
Anon 26380 103 0%
Exec and libs 1179 4 0%
Page cache 4627 18 0%
Free (cachelist) 7725 30 0%
Free (freelist) 7063409 27591 42%
Total 16766410 65493
Physical 16766409 65493
Thu Dec 4 16:15:04 IST 2014
Page Summary Pages MB %Tot
------------ ---------------- ---------------- ----
Kernel 8085951 31585 48%
ZFS File Data 1577181 6160 9%
Anon 26380 103 0%
Exec and libs 1181 4 0%
Page cache 4628 18 0%
Free (cachelist) 7723 30 0%
Free (freelist) 7063366 27591 42%
Total 16766410 65493
Physical 16766409 65493
Thu Dec 4 16:15:08 IST 2014
Page Summary Pages MB %Tot
------------ ---------------- ---------------- ----
Kernel 8209025 32066 49%
ZFS File Data 1577181 6160 9%
Anon 26407 103 0%
Exec and libs 1181 4 0%
Page cache 4630 18 0%
Free (cachelist) 7723 30 0%
Free (freelist) 6940263 27110 41%
Total 16766410 65493
Physical 16766409 65493
Thu Dec 4 16:15:11 IST 2014
Page Summary Pages MB %Tot
------------ ---------------- ---------------- ----
Kernel 8763032 34230 52%
ZFS File Data 1577181 6160 9%
Anon 26432 103 0%
Exec and libs 1181 4 0%
Page cache 4630 18 0%
Free (cachelist) 7723 30 0%
Free (freelist) 6386231 24946 38%
Total 16766410 65493
Physical 16766409 65493
Thu Dec 4 16:15:14 IST 2014
Page Summary Pages MB %Tot
------------ ---------------- ---------------- ----
Kernel 9388716 36674 56%
ZFS File Data 1577181 6160 9%
Anon 26432 103 0%
Exec and libs 1181 4 0%
Page cache 4631 18 0%
Free (cachelist) 7724 30 0%
Free (freelist) 5760545 22502 34%
Total 16766410 65493
Physical 16766409 65493
Thu Dec 4 16:15:17 IST 2014
Page Summary Pages MB %Tot
------------ ---------------- ---------------- ----
Kernel 9586189 37446 57%
ZFS File Data 1577181 6160 9%
Anon 26459 103 0%
Exec and libs 1181 4 0%
Page cache 4632 18 0%
Free (cachelist) 7723 30 0%
Free (freelist) 5563045 21730 33%
Total 16766410 65493
Physical 16766409 65493
Thu Dec 4 16:15:20 IST 2014
Page Summary Pages MB %Tot
------------ ---------------- ---------------- ----
Kernel 9967338 38934 59%
ZFS File Data 1577181 6160 9%
Anon 26459 103 0%
Exec and libs 1181 4 0%
Page cache 4633 18 0%
Free (cachelist) 7723 30 0%
Free (freelist) 5181895 20241 31%
Total 16766410 65493
Physical 16766409 65493
Thu Dec 4 16:15:24 IST 2014
Page Summary Pages MB %Tot
------------ ---------------- ---------------- ----
Kernel 10087818 39405 60%
ZFS File Data 1577181 6160 9%
Anon 26484 103 0%
Exec and libs 1181 4 0%
Page cache 4635 18 0%
Free (cachelist) 7723 30 0%
Free (freelist) 5061388 19771 30%
Total 16766410 65493
Physical 16766409 65493
Thu Dec 4 16:15:27 IST 2014
Page Summary Pages MB %Tot
------------ ---------------- ---------------- ----
Kernel 10620046 41484 63%
ZFS File Data 1577181 6160 9%
Anon 26510 103 0%
Exec and libs 1157 4 0%
Page cache 4633 18 0%
Free (cachelist) 7750 30 0%
Free (freelist) 4529133 17691 27%
Total 16766410 65493
Physical 16766409 65493
Thu Dec 4 16:15:30 IST 2014
Page Summary Pages MB %Tot
------------ ---------------- ---------------- ----
Kernel 10728161 41906 64%
ZFS File Data 1577181 6160 9%
Anon 26510 103 0%
Exec and libs 1181 4 0%
Page cache 4637 18 0%
Free (cachelist) 7723 30 0%
Free (freelist) 4421017 17269 26%
Total 16766410 65493
Physical 16766409 65493
Thu Dec 4 16:15:34 IST 2014
Page Summary Pages MB %Tot
------------ ---------------- ---------------- ----
Kernel 11230924 43870 67%
ZFS File Data 1577181 6160 9%
Anon 26536 103 0%
Exec and libs 1181 4 0%
Page cache 4637 18 0%
Free (cachelist) 7723 30 0%
Free (freelist) 3918228 15305 23%
Total 16766410 65493
Physical 16766409 65493
Thu Dec 4 16:15:38 IST 2014
Page Summary Pages MB %Tot
------------ ---------------- ---------------- ----
Kernel 11277074 44051 67%
ZFS File Data 1577981 6163 9%
Anon 26562 103 0%
Exec and libs 1181 4 0%
Page cache 4638 18 0%
Free (cachelist) 7723 30 0%
Free (freelist) 3871251 15122 23%
Total 16766410 65493
Physical 16766409 65493
Thu Dec 4 16:15:41 IST 2014
Page Summary Pages MB %Tot
------------ ---------------- ---------------- ----
Kernel 11779320 46012 70%
ZFS File Data 1577981 6163 9%
Anon 26562 103 0%
Exec and libs 1180 4 0%
Page cache 4639 18 0%
Free (cachelist) 7724 30 0%
Free (freelist) 3369004 13160 20%
Total 16766410 65493
Physical 16766409 65493
Thu Dec 4 16:15:44 IST 2014
Page Summary Pages MB %Tot
------------ ---------------- ---------------- ----
Kernel 11968305 46751 71%
ZFS File Data 1577981 6163 9%
Anon 26588 103 0%
Exec and libs 1181 4 0%
Page cache 4641 18 0%
Free (cachelist) 7723 30 0%
Free (freelist) 3179991 12421 19%
Total 16766410 65493
Physical 16766409 65493
Thu Dec 4 16:15:47 IST 2014
Page Summary Pages MB %Tot
------------ ---------------- ---------------- ----
Kernel 12424008 48531 74%
ZFS File Data 1577981 6163 9%
Anon 26614 103 0%
Exec and libs 1181 4 0%
Page cache 4642 18 0%
Free (cachelist) 7723 30 0%
Free (freelist) 2724261 10641 16%
Total 16766410 65493
Physical 16766409 65493
Thu Dec 4 16:15:51 IST 2014
Page Summary Pages MB %Tot
------------ ---------------- ---------------- ----
Kernel 12708257 49641 76%
ZFS File Data 1577981 6163 9%
Anon 26614 103 0%
Exec and libs 1181 4 0%
Page cache 4643 18 0%
Free (cachelist) 7723 30 0%
Free (freelist) 2440011 9531 15%
Total 16766410 65493
Physical 16766409 65493
Thu Dec 4 16:15:54 IST 2014
Page Summary Pages MB %Tot
------------ ---------------- ---------------- ----
Kernel 12960558 50627 77%
ZFS File Data 1577981 6163 9%
Anon 26640 104 0%
Exec and libs 1181 4 0%
Page cache 4644 18 0%
Free (cachelist) 7723 30 0%
Free (freelist) 2187683 8545 13%
Total 16766410 65493
Physical 16766409 65493
Thu Dec 4 16:15:58 IST 2014
Page Summary Pages MB %Tot
------------ ---------------- ---------------- ----
Kernel 13441434 52505 80%
ZFS File Data 1577981 6163 9%
Anon 26664 104 0%
Exec and libs 1181 4 0%
Page cache 4645 18 0%
Free (cachelist) 7723 30 0%
Free (freelist) 1706782 6667 10%
Total 16766410 65493
Physical 16766409 65493
Thu Dec 4 16:16:01 IST 2014
Page Summary Pages MB %Tot
------------ ---------------- ---------------- ----
Kernel 13724075 53609 82%
ZFS File Data 1577981 6163 9%
Anon 26691 104 0%
Exec and libs 1181 4 0%
Page cache 4646 18 0%
Free (cachelist) 7723 30 0%
Free (freelist) 1424113 5562 8%
Total 16766410 65493
Physical 16766409 65493
Thu Dec 4 16:16:05 IST 2014
Page Summary Pages MB %Tot
------------ ---------------- ---------------- ----
Kernel 14075430 54982 84%
ZFS File Data 1577981 6163 9%
Anon 26691 104 0%
Exec and libs 1181 4 0%
Page cache 4647 18 0%
Free (cachelist) 7723 30 0%
Free (freelist) 1072757 4190 6%
Total 16766410 65493
Physical 16766409 65493
Thu Dec 4 16:16:08 IST 2014
Page Summary Pages MB %Tot
------------ ---------------- ---------------- ----
Kernel 14302316 55868 85%
ZFS File Data 1577981 6163 9%
Anon 26723 104 0%
Exec and libs 1181 4 0%
Page cache 4649 18 0%
Free (cachelist) 7723 30 0%
Free (freelist) 845837 3304 5%
Total 16766410 65493
Physical 16766409 65493
Thu Dec 4 16:16:12 IST 2014
Page Summary Pages MB %Tot
------------ ---------------- ---------------- ----
Kernel 14585839 56975 87%
ZFS File Data 1541308 6020 9%
Anon 26743 104 0%
Exec and libs 1181 4 0%
Page cache 4649 18 0%
Free (cachelist) 7724 30 0%
Free (freelist) 598966 2339 4%
Total 16766410 65493
Physical 16766409 65493
Thu Dec 4 16:16:16 IST 2014
Page Summary Pages MB %Tot
------------ ---------------- ---------------- ----
Kernel 14682403 57353 88%
ZFS File Data 1408152 5500 8%
Anon 26768 104 0%
Exec and libs 1181 4 0%
Page cache 4651 18 0%
Free (cachelist) 7723 30 0%
Free (freelist) 635532 2482 4%
Total 16766410 65493
Physical 16766409 65493
Thu Dec 4 16:16:19 IST 2014
Page Summary Pages MB %Tot
------------ ---------------- ---------------- ----
Kernel 15056031 58812 90%
ZFS File Data 1163626 4545 7%
Anon 26778 104 0%
Exec and libs 1181 4 0%
Page cache 4652 18 0%
Free (cachelist) 7723 30 0%
Free (freelist) 506419 1978 3%
Total 16766410 65493
Physical 16766409 65493
Thu Dec 4 16:16:23 IST 2014
Page Summary Pages MB %Tot
------------ ---------------- ---------------- ----
Kernel 15285492 59708 91%
ZFS File Data 1008029 3937 6%
Anon 26795 104 0%
Exec and libs 1181 4 0%
Page cache 4653 18 0%
Free (cachelist) 7429 29 0%
Free (freelist) 432831 1690 3%
Total 16766410 65493
Physical 16766409 65493
Thu Dec 4 16:16:27 IST 2014
Page Summary Pages MB %Tot
------------ ---------------- ---------------- ----
Kernel 15705417 61349 94%
ZFS File Data 857418 3349 5%
Anon 21468 83 0%
Exec and libs 73 0 0%
Page cache 721 2 0%
Free (cachelist) 14622 57 0%
Free (freelist) 166691 651 1%
Total 16766410 65493
Physical 16766409 65493
-------------- next part --------------
last pid: 996; load avg: 5.95, 3.04, 1.31; up 0+00:19:43 16:16:27
59 processes: 58 sleeping, 1 on cpu
CPU states: 30.6% idle, 1.6% user, 67.8% kernel, 0.0% iowait, 0.0% swap
Kernel: 97556 ctxsw, 2440 trap, 448800 intr, 27430 syscall, 1 fork, 1407 flt
Memory: 64G phys mem, 933M free mem, 4096M total swap, 4096M free swap
PID USERNAME NLWP PRI NICE SIZE RES STATE TIME CPU COMMAND
675 daemon 10 60 -20 2904K 1968K sleep 2:28 4.35% nfsd
607 root 1 59 0 8308K 6668K sleep 0:00 0.01% perl
631 root 1 59 0 26M 25M cpu/5 0:00 0.01% top
711 root 1 59 0 5888K 4180K sleep 0:00 0.01% perl
705 root 1 59 0 4712K 2628K sleep 0:00 0.00% perl
633 root 1 59 0 7576K 4828K sleep 0:00 0.00% sshd
712 root 1 59 0 1644K 1040K sleep 0:00 0.00% tee
622 root 1 59 0 7576K 4828K sleep 0:00 0.00% sshd
645 root 1 59 0 7576K 4832K sleep 0:00 0.00% sshd
10 root 14 59 0 7684K 5664K sleep 0:01 0.00% svc.startd
580 root 27 59 0 23M 16M sleep 0:01 0.00% fmd
706 root 1 59 0 1644K 1040K sleep 0:00 0.00% tee
12 root 19 59 0 9704K 8372K sleep 0:04 0.00% svc.configd
241 root 7 59 0 4456K 3220K sleep 0:00 0.00% devfsadm
294 root 34 59 0 7480K 4936K sleep 0:00 0.00% nscd
367 root 1 59 0 1656K 940K sleep 0:00 0.00% utmpd
658 daemon 2 60 -20 2888K 1912K sleep 0:00 0.00% lockd
232 root 5 60 -20 2712K 1600K sleep 0:00 0.00% zonestatd
322 root 4 59 0 11M 8224K sleep 0:00 0.00% hald
301 root 5 59 0 5232K 4324K sleep 0:00 0.00% picld
487 root 3 59 0 5272K 3688K sleep 0:00 0.00% inetd
331 root 2 59 0 6660K 3368K sleep 0:00 0.00% hald-runner
225 root 17 59 0 6164K 3312K sleep 0:00 0.00% syseventd
632 root 1 59 0 5804K 3192K sleep 0:00 0.00% sshd
644 root 1 59 0 5804K 3192K sleep 0:00 0.00% sshd
612 root 1 59 0 6508K 3180K sleep 0:00 0.00% hald-addon-netw
614 root 1 59 0 6508K 3180K sleep 0:00 0.00% hald-addon-cpuf
621 root 1 59 0 5804K 3096K sleep 0:00 0.00% sshd
49 netadm 3 59 0 4084K 2836K sleep 0:00 0.00% ipmgmtd
673 root 13 59 0 4588K 2744K sleep 0:00 0.00% mountd
-------------- next part --------------
Thu Dec 4 16:14:39 IST 2014
l2_size = 0
other_size = 16487240
c_max = 67601473536
hash_elements = 36191
duplicate_reads = 0
prefetch_data_hits = 537
l2_read_bytes = 0
demand_metadata_misses = 12438
prefetch_data_misses = 1540
hash_chain_max = 2
l2_hits = 0
l2_compress_failures = 0
mfu_ghost_hits = 0
l2_rw_clash = 0
memory_throttle_count = 0
crtime = 4.537754192
arc_meta_used = 97172664
hash_elements_max = 41359
size = 5312841912
hits = 750510
mfu_hits = 507924
prefetch_metadata_hits = 10297
hash_chains = 54
evict_l2_cached = 0
l2_writes_sent = 0
l2_writes_error = 0
deleted = 15
hdr_size = 12559216
l2_compress_zeros = 0
l2_abort_lowmem = 0
l2_io_error = 0
l2_hdr_size = 0
l2_cksum_bad = 0
l2_evict_reading = 0
arc_meta_limit = 16900368384
prefetch_metadata_misses = 3381
l2_evict_lock_retry = 0
hash_collisions = 519
l2_writes_hdr_miss = 0
evict_l2_ineligible = 4096
l2_write_bytes = 0
data_size = 5283795456
c = 67601473536
l2_asize = 0
class = misc
mutex_miss = 0
snaptime = 1052.750004426
duplicate_buffers_size = 0
l2_misses = 0
mru_ghost_hits = 0
demand_metadata_hits = 28989
arc_meta_max = 97686696
c_min = 2146100480
mru_hits = 231752
demand_data_misses = 2615
duplicate_buffers = 0
demand_data_hits = 710687
l2_free_on_write = 0
l2_feeds = 0
misses = 19974
evict_skip = 1214
evict_l2_eligible = 71168
p = 33800736768
l2_compress_successes = 0
recycle_miss = 0
l2_writes_done = 0
----------
Thu Dec 4 16:14:42 IST 2014
l2_size = 0
other_size = 16557200
c_max = 67601473536
hash_elements = 35929
duplicate_reads = 0
prefetch_data_hits = 537
l2_read_bytes = 0
demand_metadata_misses = 12446
prefetch_data_misses = 1540
hash_chain_max = 2
l2_hits = 0
l2_compress_failures = 0
mfu_ghost_hits = 0
l2_rw_clash = 0
memory_throttle_count = 0
crtime = 4.537754192
arc_meta_used = 97848832
hash_elements_max = 41359
size = 5242345984
hits = 765628
mfu_hits = 519125
prefetch_metadata_hits = 10297
hash_chains = 53
evict_l2_cached = 0
l2_writes_sent = 0
l2_writes_error = 0
deleted = 15
hdr_size = 12559216
l2_compress_zeros = 0
l2_abort_lowmem = 0
l2_io_error = 0
l2_hdr_size = 0
l2_cksum_bad = 0
l2_evict_reading = 0
arc_meta_limit = 16900368384
prefetch_metadata_misses = 3381
l2_evict_lock_retry = 0
hash_collisions = 544
l2_writes_hdr_miss = 0
evict_l2_ineligible = 4096
l2_write_bytes = 0
data_size = 5213229568
c = 67601473536
l2_asize = 0
class = misc
mutex_miss = 0
snaptime = 1055.756679096
duplicate_buffers_size = 0
l2_misses = 0
mru_ghost_hits = 0
demand_metadata_hits = 28999
arc_meta_max = 97815864
c_min = 2146100480
mru_hits = 235669
demand_data_misses = 2615
duplicate_buffers = 0
demand_data_hits = 725795
l2_free_on_write = 0
l2_feeds = 0
misses = 19982
evict_skip = 1214
evict_l2_eligible = 71168
p = 33800736768
l2_compress_successes = 0
recycle_miss = 0
l2_writes_done = 0
----------
Thu Dec 4 16:14:45 IST 2014
l2_size = 0
other_size = 16815440
c_max = 67601473536
hash_elements = 34070
duplicate_reads = 0
prefetch_data_hits = 537
l2_read_bytes = 0
demand_metadata_misses = 12446
prefetch_data_misses = 1540
hash_chain_max = 2
l2_hits = 0
l2_compress_failures = 0
mfu_ghost_hits = 0
l2_rw_clash = 0
memory_throttle_count = 0
crtime = 4.537754192
arc_meta_used = 97619648
hash_elements_max = 41359
size = 5487483584
hits = 791669
mfu_hits = 537122
prefetch_metadata_hits = 10297
hash_chains = 50
evict_l2_cached = 0
l2_writes_sent = 0
l2_writes_error = 0
deleted = 15
hdr_size = 12559216
l2_compress_zeros = 0
l2_abort_lowmem = 0
l2_io_error = 0
l2_hdr_size = 0
l2_cksum_bad = 0
l2_evict_reading = 0
arc_meta_limit = 16900368384
prefetch_metadata_misses = 3381
l2_evict_lock_retry = 0
hash_collisions = 581
l2_writes_hdr_miss = 0
evict_l2_ineligible = 4096
l2_write_bytes = 0
data_size = 5458108928
c = 67601473536
l2_asize = 0
class = misc
mutex_miss = 0
snaptime = 1058.767796546
duplicate_buffers_size = 0
l2_misses = 0
mru_ghost_hits = 0
demand_metadata_hits = 29003
arc_meta_max = 97968040
c_min = 2146100480
mru_hits = 243713
demand_data_misses = 2615
duplicate_buffers = 0
demand_data_hits = 751832
l2_free_on_write = 0
l2_feeds = 0
misses = 19982
evict_skip = 1214
evict_l2_eligible = 71168
p = 33800736768
l2_compress_successes = 0
recycle_miss = 0
l2_writes_done = 0
----------
Thu Dec 4 16:14:48 IST 2014
l2_size = 0
other_size = 16858360
c_max = 67601473536
hash_elements = 36794
duplicate_reads = 0
prefetch_data_hits = 537
l2_read_bytes = 0
demand_metadata_misses = 12454
prefetch_data_misses = 1540
hash_chain_max = 2
l2_hits = 0
l2_compress_failures = 0
mfu_ghost_hits = 0
l2_rw_clash = 0
memory_throttle_count = 0
crtime = 4.537754192
arc_meta_used = 98313832
hash_elements_max = 41359
size = 5462880872
hits = 802717
mfu_hits = 544723
prefetch_metadata_hits = 10297
hash_chains = 75
evict_l2_cached = 0
l2_writes_sent = 0
l2_writes_error = 0
deleted = 15
hdr_size = 12559216
l2_compress_zeros = 0
l2_abort_lowmem = 0
l2_io_error = 0
l2_hdr_size = 0
l2_cksum_bad = 0
l2_evict_reading = 0
arc_meta_limit = 16900368384
prefetch_metadata_misses = 3381
l2_evict_lock_retry = 0
hash_collisions = 626
l2_writes_hdr_miss = 0
evict_l2_ineligible = 4096
l2_write_bytes = 0
data_size = 5433463296
c = 67601473536
l2_asize = 0
class = misc
mutex_miss = 0
snaptime = 1061.787519516
duplicate_buffers_size = 0
l2_misses = 0
mru_ghost_hits = 0
demand_metadata_hits = 29022
arc_meta_max = 98312872
c_min = 2146100480
mru_hits = 247160
demand_data_misses = 2615
duplicate_buffers = 0
demand_data_hits = 762861
l2_free_on_write = 0
l2_feeds = 0
misses = 19990
evict_skip = 1214
evict_l2_eligible = 71168
p = 33800736768
l2_compress_successes = 0
recycle_miss = 0
l2_writes_done = 0
----------
Thu Dec 4 16:14:51 IST 2014
l2_size = 0
other_size = 16888240
c_max = 67601473536
hash_elements = 40590
duplicate_reads = 0
prefetch_data_hits = 537
l2_read_bytes = 0
demand_metadata_misses = 12458
prefetch_data_misses = 1540
hash_chain_max = 2
l2_hits = 0
l2_compress_failures = 0
mfu_ghost_hits = 0
l2_rw_clash = 0
memory_throttle_count = 0
crtime = 4.537754192
arc_meta_used = 98425632
hash_elements_max = 42641
size = 5669168928
hits = 815255
mfu_hits = 552397
prefetch_metadata_hits = 10297
hash_chains = 86
evict_l2_cached = 0
l2_writes_sent = 0
l2_writes_error = 0
deleted = 15
hdr_size = 12559216
l2_compress_zeros = 0
l2_abort_lowmem = 0
l2_io_error = 0
l2_hdr_size = 0
l2_cksum_bad = 0
l2_evict_reading = 0
arc_meta_limit = 16900368384
prefetch_metadata_misses = 3381
l2_evict_lock_retry = 0
hash_collisions = 664
l2_writes_hdr_miss = 0
evict_l2_ineligible = 4096
l2_write_bytes = 0
data_size = 5639721472
c = 67601473536
l2_asize = 0
class = misc
mutex_miss = 0
snaptime = 1064.791199302
duplicate_buffers_size = 0
l2_misses = 0
mru_ghost_hits = 0
demand_metadata_hits = 29029
arc_meta_max = 98388464
c_min = 2146100480
mru_hits = 252024
demand_data_misses = 2615
duplicate_buffers = 0
demand_data_hits = 775392
l2_free_on_write = 0
l2_feeds = 0
misses = 19994
evict_skip = 1214
evict_l2_eligible = 71168
p = 33800736768
l2_compress_successes = 0
recycle_miss = 0
l2_writes_done = 0
----------
Thu Dec 4 16:14:54 IST 2014
l2_size = 0
other_size = 17262640
c_max = 67601473536
hash_elements = 36737
duplicate_reads = 0
prefetch_data_hits = 537
l2_read_bytes = 0
demand_metadata_misses = 12462
prefetch_data_misses = 1540
hash_chain_max = 2
l2_hits = 0
l2_compress_failures = 0
mfu_ghost_hits = 0
l2_rw_clash = 0
memory_throttle_count = 0
crtime = 4.537754192
arc_meta_used = 98304416
hash_elements_max = 42641
size = 6396890528
hits = 840144
mfu_hits = 570407
prefetch_metadata_hits = 10297
hash_chains = 80
evict_l2_cached = 0
l2_writes_sent = 0
l2_writes_error = 0
deleted = 15
hdr_size = 12559216
l2_compress_zeros = 0
l2_abort_lowmem = 0
l2_io_error = 0
l2_hdr_size = 0
l2_cksum_bad = 0
l2_evict_reading = 0
arc_meta_limit = 16900368384
prefetch_metadata_misses = 3381
l2_evict_lock_retry = 0
hash_collisions = 690
l2_writes_hdr_miss = 0
evict_l2_ineligible = 4096
l2_write_bytes = 0
data_size = 6367068672
c = 67601473536
l2_asize = 0
class = misc
mutex_miss = 0
snaptime = 1067.797220708
duplicate_buffers_size = 0
l2_misses = 0
mru_ghost_hits = 0
demand_metadata_hits = 29038
arc_meta_max = 98730168
c_min = 2146100480
mru_hits = 258903
demand_data_misses = 2615
duplicate_buffers = 0
demand_data_hits = 800272
l2_free_on_write = 0
l2_feeds = 0
misses = 19998
evict_skip = 1214
evict_l2_eligible = 71168
p = 33800736768
l2_compress_successes = 0
recycle_miss = 0
l2_writes_done = 0
----------
Thu Dec 4 16:14:57 IST 2014
l2_size = 0
other_size = 17330840
c_max = 67601473536
hash_elements = 39702
duplicate_reads = 0
prefetch_data_hits = 537
l2_read_bytes = 0
demand_metadata_misses = 12466
prefetch_data_misses = 1540
hash_chain_max = 2
l2_hits = 0
l2_compress_failures = 0
mfu_ghost_hits = 0
l2_rw_clash = 0
memory_throttle_count = 0
crtime = 4.537754192
arc_meta_used = 99036168
hash_elements_max = 42641
size = 6258292744
hits = 851791
mfu_hits = 578190
prefetch_metadata_hits = 10297
hash_chains = 63
evict_l2_cached = 0
l2_writes_sent = 0
l2_writes_error = 0
deleted = 15
hdr_size = 12559216
l2_compress_zeros = 0
l2_abort_lowmem = 0
l2_io_error = 0
l2_hdr_size = 0
l2_cksum_bad = 0
l2_evict_reading = 0
arc_meta_limit = 16900368384
prefetch_metadata_misses = 3381
l2_evict_lock_retry = 0
hash_collisions = 710
l2_writes_hdr_miss = 0
evict_l2_ineligible = 4096
l2_write_bytes = 0
data_size = 6228402688
c = 67601473536
l2_asize = 0
class = misc
mutex_miss = 0
snaptime = 1070.806236777
duplicate_buffers_size = 0
l2_misses = 0
mru_ghost_hits = 0
demand_metadata_hits = 29044
arc_meta_max = 99036168
c_min = 2146100480
mru_hits = 262767
demand_data_misses = 2615
duplicate_buffers = 0
demand_data_hits = 811913
l2_free_on_write = 0
l2_feeds = 0
misses = 20002
evict_skip = 1214
evict_l2_eligible = 71168
p = 33800736768
l2_compress_successes = 0
recycle_miss = 0
l2_writes_done = 0
----------
Thu Dec 4 16:15:00 IST 2014
l2_size = 0
other_size = 17276800
c_max = 67601473536
hash_elements = 39938
duplicate_reads = 0
prefetch_data_hits = 537
l2_read_bytes = 0
demand_metadata_misses = 12470
prefetch_data_misses = 1540
hash_chain_max = 2
l2_hits = 0
l2_compress_failures = 0
mfu_ghost_hits = 0
l2_rw_clash = 0
memory_throttle_count = 0
crtime = 4.537754192
arc_meta_used = 98994416
hash_elements_max = 42641
size = 5947348208
hits = 854591
mfu_hits = 580209
prefetch_metadata_hits = 10297
hash_chains = 64
evict_l2_cached = 0
l2_writes_sent = 0
l2_writes_error = 0
deleted = 15
hdr_size = 12559216
l2_compress_zeros = 0
l2_abort_lowmem = 0
l2_io_error = 0
l2_hdr_size = 0
l2_cksum_bad = 0
l2_evict_reading = 0
arc_meta_limit = 16900368384
prefetch_metadata_misses = 3381
l2_evict_lock_retry = 0
hash_collisions = 713
l2_writes_hdr_miss = 0
evict_l2_ineligible = 4096
l2_write_bytes = 0
data_size = 5917512192
c = 67601473536
l2_asize = 0
class = misc
mutex_miss = 0
snaptime = 1073.811610852
duplicate_buffers_size = 0
l2_misses = 0
mru_ghost_hits = 0
demand_metadata_hits = 29051
arc_meta_max = 99051168
c_min = 2146100480
mru_hits = 263548
demand_data_misses = 2615
duplicate_buffers = 0
demand_data_hits = 814706
l2_free_on_write = 0
l2_feeds = 0
misses = 20006
evict_skip = 1214
evict_l2_eligible = 71168
p = 33800736768
l2_compress_successes = 0
recycle_miss = 0
l2_writes_done = 0
----------
Thu Dec 4 16:15:03 IST 2014
l2_size = 0
other_size = 17471000
c_max = 67601473536
hash_elements = 38061
duplicate_reads = 0
prefetch_data_hits = 537
l2_read_bytes = 0
demand_metadata_misses = 12470
prefetch_data_misses = 1540
hash_chain_max = 2
l2_hits = 0
l2_compress_failures = 0
mfu_ghost_hits = 0
l2_rw_clash = 0
memory_throttle_count = 0
crtime = 4.537754192
arc_meta_used = 98656136
hash_elements_max = 42641
size = 6162492296
hits = 881460
mfu_hits = 598502
prefetch_metadata_hits = 10297
hash_chains = 77
evict_l2_cached = 0
l2_writes_sent = 0
l2_writes_error = 0
deleted = 15
hdr_size = 12559216
l2_compress_zeros = 0
l2_abort_lowmem = 0
l2_io_error = 0
l2_hdr_size = 0
l2_cksum_bad = 0
l2_evict_reading = 0
arc_meta_limit = 16900368384
prefetch_metadata_misses = 3381
l2_evict_lock_retry = 0
hash_collisions = 756
l2_writes_hdr_miss = 0
evict_l2_ineligible = 4096
l2_write_bytes = 0
data_size = 6132462080
c = 67601473536
l2_asize = 0
class = misc
mutex_miss = 0
snaptime = 1076.82642884
duplicate_buffers_size = 0
l2_misses = 0
mru_ghost_hits = 0
demand_metadata_hits = 29052
arc_meta_max = 99051168
c_min = 2146100480
mru_hits = 272124
demand_data_misses = 2615
duplicate_buffers = 0
demand_data_hits = 841574
l2_free_on_write = 0
l2_feeds = 0
misses = 20006
evict_skip = 1214
evict_l2_eligible = 71168
p = 33800736768
l2_compress_successes = 0
recycle_miss = 0
l2_writes_done = 0
----------
Thu Dec 4 16:15:06 IST 2014
l2_size = 0
other_size = 17531840
c_max = 67601473536
hash_elements = 38754
duplicate_reads = 0
prefetch_data_hits = 537
l2_read_bytes = 0
demand_metadata_misses = 12478
prefetch_data_misses = 1540
hash_chain_max = 2
l2_hits = 0
l2_compress_failures = 0
mfu_ghost_hits = 0
l2_rw_clash = 0
memory_throttle_count = 0
crtime = 4.537754192
arc_meta_used = 98766128
hash_elements_max = 42641
size = 6131800368
hits = 891251
mfu_hits = 605449
prefetch_metadata_hits = 10297
hash_chains = 83
evict_l2_cached = 0
l2_writes_sent = 0
l2_writes_error = 0
deleted = 15
hdr_size = 12559216
l2_compress_zeros = 0
l2_abort_lowmem = 0
l2_io_error = 0
l2_hdr_size = 0
l2_cksum_bad = 0
l2_evict_reading = 0
arc_meta_limit = 16900368384
prefetch_metadata_misses = 3381
l2_evict_lock_retry = 0
hash_collisions = 774
l2_writes_hdr_miss = 0
evict_l2_ineligible = 4096
l2_write_bytes = 0
data_size = 6101709312
c = 67601473536
l2_asize = 0
class = misc
mutex_miss = 0
snaptime = 1079.87068534
duplicate_buffers_size = 0
l2_misses = 0
mru_ghost_hits = 0
demand_metadata_hits = 29066
arc_meta_max = 99360176
c_min = 2146100480
mru_hits = 274968
demand_data_misses = 2615
duplicate_buffers = 0
demand_data_hits = 851351
l2_free_on_write = 0
l2_feeds = 0
misses = 20014
evict_skip = 1214
evict_l2_eligible = 71168
p = 33800736768
l2_compress_successes = 0
recycle_miss = 0
l2_writes_done = 0
----------
Thu Dec 4 16:15:09 IST 2014
l2_size = 0
other_size = 17593040
c_max = 67601473536
hash_elements = 40255
duplicate_reads = 0
prefetch_data_hits = 537
l2_read_bytes = 0
demand_metadata_misses = 12478
prefetch_data_misses = 1540
hash_chain_max = 2
l2_hits = 0
l2_compress_failures = 0
mfu_ghost_hits = 0
l2_rw_clash = 0
memory_throttle_count = 0
crtime = 4.537754192
arc_meta_used = 98860096
hash_elements_max = 42641
size = 6205425728
hits = 902579
mfu_hits = 612627
prefetch_metadata_hits = 10297
hash_chains = 81
evict_l2_cached = 0
l2_writes_sent = 0
l2_writes_error = 0
deleted = 15
hdr_size = 12559216
l2_compress_zeros = 0
l2_abort_lowmem = 0
l2_io_error = 0
l2_hdr_size = 0
l2_cksum_bad = 0
l2_evict_reading = 0
arc_meta_limit = 16900368384
prefetch_metadata_misses = 3381
l2_evict_lock_retry = 0
hash_collisions = 794
l2_writes_hdr_miss = 0
evict_l2_ineligible = 4096
l2_write_bytes = 0
data_size = 6175273472
c = 67601473536
l2_asize = 0
class = misc
mutex_miss = 0
snaptime = 1082.875826687
duplicate_buffers_size = 0
l2_misses = 0
mru_ghost_hits = 0
demand_metadata_hits = 29066
arc_meta_max = 99360176
c_min = 2146100480
mru_hits = 279118
demand_data_misses = 2615
duplicate_buffers = 0
demand_data_hits = 862679
l2_free_on_write = 0
l2_feeds = 0
misses = 20014
evict_skip = 1214
evict_l2_eligible = 71168
p = 33800736768
l2_compress_successes = 0
recycle_miss = 0
l2_writes_done = 0
----------
Thu Dec 4 16:15:12 IST 2014
l2_size = 0
other_size = 17598240
c_max = 67601473536
hash_elements = 43117
duplicate_reads = 0
prefetch_data_hits = 537
l2_read_bytes = 0
demand_metadata_misses = 12478
prefetch_data_misses = 1540
hash_chain_max = 2
l2_hits = 0
l2_compress_failures = 0
mfu_ghost_hits = 0
l2_rw_clash = 0
memory_throttle_count = 0
crtime = 4.537754192
arc_meta_used = 98865296
hash_elements_max = 43118
size = 6013541520
hits = 905362
mfu_hits = 613511
prefetch_metadata_hits = 10297
hash_chains = 111
evict_l2_cached = 0
l2_writes_sent = 0
l2_writes_error = 0
deleted = 15
hdr_size = 12559216
l2_compress_zeros = 0
l2_abort_lowmem = 0
l2_io_error = 0
l2_hdr_size = 0
l2_cksum_bad = 0
l2_evict_reading = 0
arc_meta_limit = 16900368384
prefetch_metadata_misses = 3381
l2_evict_lock_retry = 0
hash_collisions = 831
l2_writes_hdr_miss = 0
evict_l2_ineligible = 4096
l2_write_bytes = 0
data_size = 5983384064
c = 67601473536
l2_asize = 0
class = misc
mutex_miss = 0
snaptime = 1085.879514131
duplicate_buffers_size = 0
l2_misses = 0
mru_ghost_hits = 0
demand_metadata_hits = 29066
arc_meta_max = 99360176
c_min = 2146100480
mru_hits = 281017
demand_data_misses = 2615
duplicate_buffers = 0
demand_data_hits = 865462
l2_free_on_write = 0
l2_feeds = 0
misses = 20014
evict_skip = 1214
evict_l2_eligible = 71168
p = 33800736768
l2_compress_successes = 0
recycle_miss = 0
l2_writes_done = 0
----------
Thu Dec 4 16:15:15 IST 2014
l2_size = 0
other_size = 17603440
c_max = 67601473536
hash_elements = 43174
duplicate_reads = 0
prefetch_data_hits = 537
l2_read_bytes = 0
demand_metadata_misses = 12482
prefetch_data_misses = 1540
hash_chain_max = 2
l2_hits = 0
l2_compress_failures = 0
mfu_ghost_hits = 0
l2_rw_clash = 0
memory_throttle_count = 0
crtime = 4.537754192
arc_meta_used = 99480800
hash_elements_max = 43929
size = 5807325408
hits = 908319
mfu_hits = 615797
prefetch_metadata_hits = 10297
hash_chains = 102
evict_l2_cached = 0
l2_writes_sent = 0
l2_writes_error = 0
deleted = 15
hdr_size = 12559216
l2_compress_zeros = 0
l2_abort_lowmem = 0
l2_io_error = 0
l2_hdr_size = 0
l2_cksum_bad = 0
l2_evict_reading = 0
arc_meta_limit = 16900368384
prefetch_metadata_misses = 3381
l2_evict_lock_retry = 0
hash_collisions = 832
l2_writes_hdr_miss = 0
evict_l2_ineligible = 4096
l2_write_bytes = 0
data_size = 5777162752
c = 67601473536
l2_asize = 0
class = misc
mutex_miss = 0
snaptime = 1088.884358936
duplicate_buffers_size = 0
l2_misses = 0
mru_ghost_hits = 0
demand_metadata_hits = 29075
arc_meta_max = 99480800
c_min = 2146100480
mru_hits = 281688
demand_data_misses = 2615
duplicate_buffers = 0
demand_data_hits = 868410
l2_free_on_write = 0
l2_feeds = 0
misses = 20018
evict_skip = 1214
evict_l2_eligible = 71168
p = 33800736768
l2_compress_successes = 0
recycle_miss = 0
l2_writes_done = 0
----------
Thu Dec 4 16:15:18 IST 2014
l2_size = 0
other_size = 17668200
c_max = 67601473536
hash_elements = 41509
duplicate_reads = 0
prefetch_data_hits = 537
l2_read_bytes = 0
demand_metadata_misses = 12488
prefetch_data_misses = 1540
hash_chain_max = 2
l2_hits = 0
l2_compress_failures = 0
mfu_ghost_hits = 0
l2_rw_clash = 0
memory_throttle_count = 0
crtime = 4.537754192
arc_meta_used = 98984408
hash_elements_max = 43929
size = 5884816856
hits = 927065
mfu_hits = 630052
prefetch_metadata_hits = 10297
hash_chains = 93
evict_l2_cached = 0
l2_writes_sent = 0
l2_writes_error = 0
deleted = 15
hdr_size = 12559216
l2_compress_zeros = 0
l2_abort_lowmem = 0
l2_io_error = 0
l2_hdr_size = 0
l2_cksum_bad = 0
l2_evict_reading = 0
arc_meta_limit = 16900368384
prefetch_metadata_misses = 3381
l2_evict_lock_retry = 0
hash_collisions = 843
l2_writes_hdr_miss = 0
evict_l2_ineligible = 4096
l2_write_bytes = 0
data_size = 5854589440
c = 67601473536
l2_asize = 0
class = misc
mutex_miss = 0
snaptime = 1091.888514904
duplicate_buffers_size = 0
l2_misses = 0
mru_ghost_hits = 0
demand_metadata_hits = 29087
arc_meta_max = 99568328
c_min = 2146100480
mru_hits = 286179
demand_data_misses = 2615
duplicate_buffers = 0
demand_data_hits = 887144
l2_free_on_write = 0
l2_feeds = 0
misses = 20024
evict_skip = 1214
evict_l2_eligible = 71168
p = 33800736768
l2_compress_successes = 0
recycle_miss = 0
l2_writes_done = 0
----------
Thu Dec 4 16:15:21 IST 2014
l2_size = 0
other_size = 17646400
c_max = 67601473536
hash_elements = 39031
duplicate_reads = 0
prefetch_data_hits = 537
l2_read_bytes = 0
demand_metadata_misses = 12488
prefetch_data_misses = 1540
hash_chain_max = 2
l2_hits = 0
l2_compress_failures = 0
mfu_ghost_hits = 0
l2_rw_clash = 0
memory_throttle_count = 0
crtime = 4.537754192
arc_meta_used = 98978992
hash_elements_max = 43929
size = 5865412784
hits = 955131
mfu_hits = 649821
prefetch_metadata_hits = 10297
hash_chains = 76
evict_l2_cached = 0
l2_writes_sent = 0
l2_writes_error = 0
deleted = 15
hdr_size = 12559216
l2_compress_zeros = 0
l2_abort_lowmem = 0
l2_io_error = 0
l2_hdr_size = 0
l2_cksum_bad = 0
l2_evict_reading = 0
arc_meta_limit = 16900368384
prefetch_metadata_misses = 3381
l2_evict_lock_retry = 0
hash_collisions = 857
l2_writes_hdr_miss = 0
evict_l2_ineligible = 4096
l2_write_bytes = 0
data_size = 5835207168
c = 67601473536
l2_asize = 0
class = misc
mutex_miss = 0
snaptime = 1094.945818674
duplicate_buffers_size = 0
l2_misses = 0
mru_ghost_hits = 0
demand_metadata_hits = 29087
arc_meta_max = 99568328
c_min = 2146100480
mru_hits = 294476
demand_data_misses = 2615
duplicate_buffers = 0
demand_data_hits = 915210
l2_free_on_write = 0
l2_feeds = 0
misses = 20024
evict_skip = 1214
evict_l2_eligible = 71168
p = 33800736768
l2_compress_successes = 0
recycle_miss = 0
l2_writes_done = 0
----------
Thu Dec 4 16:15:24 IST 2014
l2_size = 0
other_size = 17752040
c_max = 67601473536
hash_elements = 41863
duplicate_reads = 0
prefetch_data_hits = 537
l2_read_bytes = 0
demand_metadata_misses = 12492
prefetch_data_misses = 1540
hash_chain_max = 2
l2_hits = 0
l2_compress_failures = 0
mfu_ghost_hits = 0
l2_rw_clash = 0
memory_throttle_count = 0
crtime = 4.537754192
arc_meta_used = 99137880
hash_elements_max = 43929
size = 6192858456
hits = 968809
mfu_hits = 659725
prefetch_metadata_hits = 10297
hash_chains = 87
evict_l2_cached = 0
l2_writes_sent = 0
l2_writes_error = 0
deleted = 15
hdr_size = 12559216
l2_compress_zeros = 0
l2_abort_lowmem = 0
l2_io_error = 0
l2_hdr_size = 0
l2_cksum_bad = 0
l2_evict_reading = 0
arc_meta_limit = 16900368384
prefetch_metadata_misses = 3381
l2_evict_lock_retry = 0
hash_collisions = 880
l2_writes_hdr_miss = 0
evict_l2_ineligible = 4096
l2_write_bytes = 0
data_size = 6162547200
c = 67601473536
l2_asize = 0
class = misc
mutex_miss = 0
snaptime = 1097.957144074
duplicate_buffers_size = 0
l2_misses = 0
mru_ghost_hits = 0
demand_metadata_hits = 29094
arc_meta_max = 99660136
c_min = 2146100480
mru_hits = 298250
demand_data_misses = 2615
duplicate_buffers = 0
demand_data_hits = 928881
l2_free_on_write = 0
l2_feeds = 0
misses = 20028
evict_skip = 1214
evict_l2_eligible = 71168
p = 33800736768
l2_compress_successes = 0
recycle_miss = 0
l2_writes_done = 0
----------
Thu Dec 4 16:15:27 IST 2014
l2_size = 0
other_size = 17651840
c_max = 67601473536
hash_elements = 44052
duplicate_reads = 0
prefetch_data_hits = 537
l2_read_bytes = 0
demand_metadata_misses = 12496
prefetch_data_misses = 1540
hash_chain_max = 2
l2_hits = 0
l2_compress_failures = 0
mfu_ghost_hits = 0
l2_rw_clash = 0
memory_throttle_count = 0
crtime = 4.537754192
arc_meta_used = 99623408
hash_elements_max = 44527
size = 5948239344
hits = 972940
mfu_hits = 661821
prefetch_metadata_hits = 10297
hash_chains = 92
evict_l2_cached = 0
l2_writes_sent = 0
l2_writes_error = 0
deleted = 15
hdr_size = 12559216
l2_compress_zeros = 0
l2_abort_lowmem = 0
l2_io_error = 0
l2_hdr_size = 0
l2_cksum_bad = 0
l2_evict_reading = 0
arc_meta_limit = 16900368384
prefetch_metadata_misses = 3381
l2_evict_lock_retry = 0
hash_collisions = 891
l2_writes_hdr_miss = 0
evict_l2_ineligible = 4096
l2_write_bytes = 0
data_size = 5918028288
c = 67601473536
l2_asize = 0
class = misc
mutex_miss = 0
snaptime = 1100.964341965
duplicate_buffers_size = 0
l2_misses = 0
mru_ghost_hits = 0
demand_metadata_hits = 29100
arc_meta_max = 99660136
c_min = 2146100480
mru_hits = 300285
demand_data_misses = 2615
duplicate_buffers = 0
demand_data_hits = 933006
l2_free_on_write = 0
l2_feeds = 0
misses = 20032
evict_skip = 1214
evict_l2_eligible = 71168
p = 33800736768
l2_compress_successes = 0
recycle_miss = 0
l2_writes_done = 0
----------
Thu Dec 4 16:15:30 IST 2014
l2_size = 0
other_size = 17736840
c_max = 67601473536
hash_elements = 39286
duplicate_reads = 0
prefetch_data_hits = 537
l2_read_bytes = 0
demand_metadata_misses = 12500
prefetch_data_misses = 1540
hash_chain_max = 2
l2_hits = 0
l2_compress_failures = 0
mfu_ghost_hits = 0
l2_rw_clash = 0
memory_throttle_count = 0
crtime = 4.537754192
arc_meta_used = 99134968
hash_elements_max = 44527
size = 5866748408
hits = 999296
mfu_hits = 681359
prefetch_metadata_hits = 10297
hash_chains = 89
evict_l2_cached = 0
l2_writes_sent = 0
l2_writes_error = 0
deleted = 15
hdr_size = 12559216
l2_compress_zeros = 0
l2_abort_lowmem = 0
l2_io_error = 0
l2_hdr_size = 0
l2_cksum_bad = 0
l2_evict_reading = 0
arc_meta_limit = 16900368384
prefetch_metadata_misses = 3381
l2_evict_lock_retry = 0
hash_collisions = 922
l2_writes_hdr_miss = 0
evict_l2_ineligible = 4096
l2_write_bytes = 0
data_size = 5836452352
c = 67601473536
l2_asize = 0
class = misc
mutex_miss = 0
snaptime = 1103.989100171
duplicate_buffers_size = 0
l2_misses = 0
mru_ghost_hits = 0
demand_metadata_hits = 29108
arc_meta_max = 99718496
c_min = 2146100480
mru_hits = 307103
demand_data_misses = 2615
duplicate_buffers = 0
demand_data_hits = 959354
l2_free_on_write = 0
l2_feeds = 0
misses = 20036
evict_skip = 1214
evict_l2_eligible = 71168
p = 33800736768
l2_compress_successes = 0
recycle_miss = 0
l2_writes_done = 0
----------
Thu Dec 4 16:15:33 IST 2014
l2_size = 0
other_size = 17708240
c_max = 67601473536
hash_elements = 43427
duplicate_reads = 0
prefetch_data_hits = 537
l2_read_bytes = 0
demand_metadata_misses = 12500
prefetch_data_misses = 1540
hash_chain_max = 2
l2_hits = 0
l2_compress_failures = 0
mfu_ghost_hits = 0
l2_rw_clash = 0
memory_throttle_count = 0
crtime = 4.537754192
arc_meta_used = 99106368
hash_elements_max = 44527
size = 5953882688
hits = 1009876
mfu_hits = 687985
prefetch_metadata_hits = 10297
hash_chains = 107
evict_l2_cached = 0
l2_writes_sent = 0
l2_writes_error = 0
deleted = 15
hdr_size = 12559216
l2_compress_zeros = 0
l2_abort_lowmem = 0
l2_io_error = 0
l2_hdr_size = 0
l2_cksum_bad = 0
l2_evict_reading = 0
arc_meta_limit = 16900368384
prefetch_metadata_misses = 3381
l2_evict_lock_retry = 0
hash_collisions = 959
l2_writes_hdr_miss = 0
evict_l2_ineligible = 4096
l2_write_bytes = 0
data_size = 5923615232
c = 67601473536
l2_asize = 0
class = misc
mutex_miss = 0
snaptime = 1106.99462468
duplicate_buffers_size = 0
l2_misses = 0
mru_ghost_hits = 0
demand_metadata_hits = 29109
arc_meta_max = 99718496
c_min = 2146100480
mru_hits = 311057
demand_data_misses = 2615
duplicate_buffers = 0
demand_data_hits = 969933
l2_free_on_write = 0
l2_feeds = 0
misses = 20036
evict_skip = 1214
evict_l2_eligible = 71168
p = 33800736768
l2_compress_successes = 0
recycle_miss = 0
l2_writes_done = 0
----------
Thu Dec 4 16:15:36 IST 2014
l2_size = 0
other_size = 17820440
c_max = 67601473536
hash_elements = 38756
duplicate_reads = 0
prefetch_data_hits = 537
l2_read_bytes = 0
demand_metadata_misses = 12508
prefetch_data_misses = 1540
hash_chain_max = 2
l2_hits = 0
l2_compress_failures = 0
mfu_ghost_hits = 0
l2_rw_clash = 0
memory_throttle_count = 0
crtime = 4.537754192
arc_meta_used = 99222664
hash_elements_max = 44527
size = 5931061384
hits = 1034158
mfu_hits = 705727
prefetch_metadata_hits = 10297
hash_chains = 79
evict_l2_cached = 0
l2_writes_sent = 0
l2_writes_error = 0
deleted = 15
hdr_size = 12559216
l2_compress_zeros = 0
l2_abort_lowmem = 0
l2_io_error = 0
l2_hdr_size = 0
l2_cksum_bad = 0
l2_evict_reading = 0
arc_meta_limit = 16900368384
prefetch_metadata_misses = 3381
l2_evict_lock_retry = 0
hash_collisions = 970
l2_writes_hdr_miss = 0
evict_l2_ineligible = 4096
l2_write_bytes = 0
data_size = 5900681728
c = 67601473536
l2_asize = 0
class = misc
mutex_miss = 0
snaptime = 1109.998397046
duplicate_buffers_size = 0
l2_misses = 0
mru_ghost_hits = 0
demand_metadata_hits = 29126
arc_meta_max = 99816784
c_min = 2146100480
mru_hits = 317597
demand_data_misses = 2615
duplicate_buffers = 0
demand_data_hits = 994198
l2_free_on_write = 0
l2_feeds = 0
misses = 20044
evict_skip = 1214
evict_l2_eligible = 71168
p = 33800736768
l2_compress_successes = 0
recycle_miss = 0
l2_writes_done = 0
----------
Thu Dec 4 16:15:39 IST 2014
l2_size = 0
other_size = 17808640
c_max = 67601473536
hash_elements = 44706
duplicate_reads = 0
prefetch_data_hits = 537
l2_read_bytes = 0
demand_metadata_misses = 12508
prefetch_data_misses = 1540
hash_chain_max = 2
l2_hits = 0
l2_compress_failures = 0
mfu_ghost_hits = 0
l2_rw_clash = 0
memory_throttle_count = 0
crtime = 4.537754192
arc_meta_used = 99212592
hash_elements_max = 44706
size = 6196209968
hits = 1039940
mfu_hits = 709638
prefetch_metadata_hits = 10297
hash_chains = 103
evict_l2_cached = 0
l2_writes_sent = 0
l2_writes_error = 0
deleted = 15
hdr_size = 12560944
l2_compress_zeros = 0
l2_abort_lowmem = 0
l2_io_error = 0
l2_hdr_size = 0
l2_cksum_bad = 0
l2_evict_reading = 0
arc_meta_limit = 16900368384
prefetch_metadata_misses = 3381
l2_evict_lock_retry = 0
hash_collisions = 1014
l2_writes_hdr_miss = 0
evict_l2_ineligible = 4096
l2_write_bytes = 0
data_size = 6165840384
c = 67601473536
l2_asize = 0
class = misc
mutex_miss = 0
snaptime = 1113.002157194
duplicate_buffers_size = 0
l2_misses = 0
mru_ghost_hits = 0
demand_metadata_hits = 29126
arc_meta_max = 99816784
c_min = 2146100480
mru_hits = 319468
demand_data_misses = 2615
duplicate_buffers = 0
demand_data_hits = 999980
l2_free_on_write = 0
l2_feeds = 0
misses = 20044
evict_skip = 1214
evict_l2_eligible = 71168
p = 33800736768
l2_compress_successes = 0
recycle_miss = 0
l2_writes_done = 0
----------
Thu Dec 4 16:15:42 IST 2014
l2_size = 0
other_size = 17820240
c_max = 67601473536
hash_elements = 42808
duplicate_reads = 0
prefetch_data_hits = 537
l2_read_bytes = 0
demand_metadata_misses = 12516
prefetch_data_misses = 1540
hash_chain_max = 2
l2_hits = 0
l2_compress_failures = 0
mfu_ghost_hits = 0
l2_rw_clash = 0
memory_throttle_count = 0
crtime = 4.537754192
arc_meta_used = 99240576
hash_elements_max = 46675
size = 6415390336
hits = 1059255
mfu_hits = 723283
prefetch_metadata_hits = 10297
hash_chains = 108
evict_l2_cached = 0
l2_writes_sent = 0
l2_writes_error = 0
deleted = 15
hdr_size = 12560944
l2_compress_zeros = 0
l2_abort_lowmem = 0
l2_io_error = 0
l2_hdr_size = 0
l2_cksum_bad = 0
l2_evict_reading = 0
arc_meta_limit = 16900368384
prefetch_metadata_misses = 3381
l2_evict_lock_retry = 0
hash_collisions = 1042
l2_writes_hdr_miss = 0
evict_l2_ineligible = 4096
l2_write_bytes = 0
data_size = 6385009152
c = 67601473536
l2_asize = 0
class = misc
mutex_miss = 0
snaptime = 1116.00772062
duplicate_buffers_size = 0
l2_misses = 0
mru_ghost_hits = 0
demand_metadata_hits = 29140
arc_meta_max = 99824704
c_min = 2146100480
mru_hits = 325138
demand_data_misses = 2615
duplicate_buffers = 0
demand_data_hits = 1019281
l2_free_on_write = 0
l2_feeds = 0
misses = 20052
evict_skip = 1214
evict_l2_eligible = 71168
p = 33800736768
l2_compress_successes = 0
recycle_miss = 0
l2_writes_done = 0
----------
Thu Dec 4 16:15:45 IST 2014
l2_size = 0
other_size = 17674040
c_max = 67601473536
hash_elements = 44891
duplicate_reads = 0
prefetch_data_hits = 537
l2_read_bytes = 0
demand_metadata_misses = 12516
prefetch_data_misses = 1540
hash_chain_max = 2
l2_hits = 0
l2_compress_failures = 0
mfu_ghost_hits = 0
l2_rw_clash = 0
memory_throttle_count = 0
crtime = 4.537754192
arc_meta_used = 99094376
hash_elements_max = 46675
size = 5900262248
hits = 1068199
mfu_hits = 728474
prefetch_metadata_hits = 10297
hash_chains = 130
evict_l2_cached = 0
l2_writes_sent = 0
l2_writes_error = 0
deleted = 15
hdr_size = 12560944
l2_compress_zeros = 0
l2_abort_lowmem = 0
l2_io_error = 0
l2_hdr_size = 0
l2_cksum_bad = 0
l2_evict_reading = 0
arc_meta_limit = 16900368384
prefetch_metadata_misses = 3381
l2_evict_lock_retry = 0
hash_collisions = 1081
l2_writes_hdr_miss = 0
evict_l2_ineligible = 4096
l2_write_bytes = 0
data_size = 5870027264
c = 67601473536
l2_asize = 0
class = misc
mutex_miss = 0
snaptime = 1119.021235208
duplicate_buffers_size = 0
l2_misses = 0
mru_ghost_hits = 0
demand_metadata_hits = 29140
arc_meta_max = 99824704
c_min = 2146100480
mru_hits = 328891
demand_data_misses = 2615
duplicate_buffers = 0
demand_data_hits = 1028225
l2_free_on_write = 0
l2_feeds = 0
misses = 20052
evict_skip = 1214
evict_l2_eligible = 71168
p = 33800736768
l2_compress_successes = 0
recycle_miss = 0
l2_writes_done = 0
----------
Thu Dec 4 16:15:48 IST 2014
l2_size = 0
other_size = 17766200
c_max = 67601473536
hash_elements = 39994
duplicate_reads = 0
prefetch_data_hits = 537
l2_read_bytes = 0
demand_metadata_misses = 12524
prefetch_data_misses = 1540
hash_chain_max = 2
l2_hits = 0
l2_compress_failures = 0
mfu_ghost_hits = 0
l2_rw_clash = 0
memory_throttle_count = 0
crtime = 4.537754192
arc_meta_used = 99751784
hash_elements_max = 46675
size = 5771420520
hits = 1088758
mfu_hits = 743653
prefetch_metadata_hits = 10297
hash_chains = 94
evict_l2_cached = 0
l2_writes_sent = 0
l2_writes_error = 0
deleted = 15
hdr_size = 12560944
l2_compress_zeros = 0
l2_abort_lowmem = 0
l2_io_error = 0
l2_hdr_size = 0
l2_cksum_bad = 0
l2_evict_reading = 0
arc_meta_limit = 16900368384
prefetch_metadata_misses = 3381
l2_evict_lock_retry = 0
hash_collisions = 1092
l2_writes_hdr_miss = 0
evict_l2_ineligible = 4096
l2_write_bytes = 0
data_size = 5741093376
c = 67601473536
l2_asize = 0
class = misc
mutex_miss = 0
snaptime = 1122.026835395
duplicate_buffers_size = 0
l2_misses = 0
mru_ghost_hits = 0
demand_metadata_hits = 29147
arc_meta_max = 99824704
c_min = 2146100480
mru_hits = 334271
demand_data_misses = 2615
duplicate_buffers = 0
demand_data_hits = 1048777
l2_free_on_write = 0
l2_feeds = 0
misses = 20060
evict_skip = 1214
evict_l2_eligible = 71168
p = 33800736768
l2_compress_successes = 0
recycle_miss = 0
l2_writes_done = 0
----------
Thu Dec 4 16:15:51 IST 2014
l2_size = 0
other_size = 17829240
c_max = 67601473536
hash_elements = 43380
duplicate_reads = 0
prefetch_data_hits = 537
l2_read_bytes = 0
demand_metadata_misses = 12524
prefetch_data_misses = 1540
hash_chain_max = 2
l2_hits = 0
l2_compress_failures = 0
mfu_ghost_hits = 0
l2_rw_clash = 0
memory_throttle_count = 0
crtime = 4.537754192
arc_meta_used = 99249576
hash_elements_max = 46675
size = 6035421608
hits = 1096297
mfu_hits = 749091
prefetch_metadata_hits = 10297
hash_chains = 87
evict_l2_cached = 0
l2_writes_sent = 0
l2_writes_error = 0
deleted = 15
hdr_size = 12560944
l2_compress_zeros = 0
l2_abort_lowmem = 0
l2_io_error = 0
l2_hdr_size = 0
l2_cksum_bad = 0
l2_evict_reading = 0
arc_meta_limit = 16900368384
prefetch_metadata_misses = 3381
l2_evict_lock_retry = 0
hash_collisions = 1102
l2_writes_hdr_miss = 0
evict_l2_ineligible = 4096
l2_write_bytes = 0
data_size = 6005031424
c = 67601473536
l2_asize = 0
class = misc
mutex_miss = 0
snaptime = 1125.031042354
duplicate_buffers_size = 0
l2_misses = 0
mru_ghost_hits = 0
demand_metadata_hits = 29154
arc_meta_max = 99829992
c_min = 2146100480
mru_hits = 336372
demand_data_misses = 2615
duplicate_buffers = 0
demand_data_hits = 1056309
l2_free_on_write = 0
l2_feeds = 0
misses = 20060
evict_skip = 1214
evict_l2_eligible = 71168
p = 33800736768
l2_compress_successes = 0
recycle_miss = 0
l2_writes_done = 0
----------
Thu Dec 4 16:15:54 IST 2014
l2_size = 0
other_size = 17900600
c_max = 67601473536
hash_elements = 39997
duplicate_reads = 0
prefetch_data_hits = 537
l2_read_bytes = 0
demand_metadata_misses = 12528
prefetch_data_misses = 1540
hash_chain_max = 2
l2_hits = 0
l2_compress_failures = 0
mfu_ghost_hits = 0
l2_rw_clash = 0
memory_throttle_count = 0
crtime = 4.537754192
arc_meta_used = 99390568
hash_elements_max = 46675
size = 5948399720
hits = 1120591
mfu_hits = 765902
prefetch_metadata_hits = 10297
hash_chains = 83
evict_l2_cached = 0
l2_writes_sent = 0
l2_writes_error = 0
deleted = 15
hdr_size = 12560944
l2_compress_zeros = 0
l2_abort_lowmem = 0
l2_io_error = 0
l2_hdr_size = 0
l2_cksum_bad = 0
l2_evict_reading = 0
arc_meta_limit = 16900368384
prefetch_metadata_misses = 3381
l2_evict_lock_retry = 0
hash_collisions = 1118
l2_writes_hdr_miss = 0
evict_l2_ineligible = 4096
l2_write_bytes = 0
data_size = 5917938176
c = 67601473536
l2_asize = 0
class = misc
mutex_miss = 0
snaptime = 1128.034852134
duplicate_buffers_size = 0
l2_misses = 0
mru_ghost_hits = 0
demand_metadata_hits = 29163
arc_meta_max = 99971728
c_min = 2146100480
mru_hits = 343855
demand_data_misses = 2615
duplicate_buffers = 0
demand_data_hits = 1080594
l2_free_on_write = 0
l2_feeds = 0
misses = 20064
evict_skip = 1214
evict_l2_eligible = 71168
p = 33800736768
l2_compress_successes = 0
recycle_miss = 0
l2_writes_done = 0
----------
Thu Dec 4 16:15:57 IST 2014
l2_size = 0
other_size = 17931600
c_max = 67601473536
hash_elements = 44425
duplicate_reads = 0
prefetch_data_hits = 537
l2_read_bytes = 0
demand_metadata_misses = 12532
prefetch_data_misses = 1540
hash_chain_max = 2
l2_hits = 0
l2_compress_failures = 0
mfu_ghost_hits = 0
l2_rw_clash = 0
memory_throttle_count = 0
crtime = 4.537754192
arc_meta_used = 100027776
hash_elements_max = 46675
size = 5889137024
hits = 1123794
mfu_hits = 768068
prefetch_metadata_hits = 10297
hash_chains = 102
evict_l2_cached = 0
l2_writes_sent = 0
l2_writes_error = 0
deleted = 15
hdr_size = 12560944
l2_compress_zeros = 0
l2_abort_lowmem = 0
l2_io_error = 0
l2_hdr_size = 0
l2_cksum_bad = 0
l2_evict_reading = 0
arc_meta_limit = 16900368384
prefetch_metadata_misses = 3381
l2_evict_lock_retry = 0
hash_collisions = 1137
l2_writes_hdr_miss = 0
evict_l2_ineligible = 4096
l2_write_bytes = 0
data_size = 5858644480
c = 67601473536
l2_asize = 0
class = misc
mutex_miss = 0
snaptime = 1131.040711699
duplicate_buffers_size = 0
l2_misses = 0
mru_ghost_hits = 0
demand_metadata_hits = 29170
arc_meta_max = 99994208
c_min = 2146100480
mru_hits = 344892
demand_data_misses = 2615
duplicate_buffers = 0
demand_data_hits = 1083790
l2_free_on_write = 0
l2_feeds = 0
misses = 20068
evict_skip = 1214
evict_l2_eligible = 71168
p = 33800736768
l2_compress_successes = 0
recycle_miss = 0
l2_writes_done = 0
----------
Thu Dec 4 16:16:00 IST 2014
l2_size = 0
other_size = 17803560
c_max = 67601473536
hash_elements = 42523
duplicate_reads = 0
prefetch_data_hits = 537
l2_read_bytes = 0
demand_metadata_misses = 12532
prefetch_data_misses = 1540
hash_chain_max = 2
l2_hits = 0
l2_compress_failures = 0
mfu_ghost_hits = 0
l2_rw_clash = 0
memory_throttle_count = 0
crtime = 4.537754192
arc_meta_used = 99334488
hash_elements_max = 46675
size = 5995791704
hits = 1137123
mfu_hits = 776941
prefetch_metadata_hits = 10297
hash_chains = 84
evict_l2_cached = 0
l2_writes_sent = 0
l2_writes_error = 0
deleted = 15
hdr_size = 12560944
l2_compress_zeros = 0
l2_abort_lowmem = 0
l2_io_error = 0
l2_hdr_size = 0
l2_cksum_bad = 0
l2_evict_reading = 0
arc_meta_limit = 16900368384
prefetch_metadata_misses = 3381
l2_evict_lock_retry = 0
hash_collisions = 1145
l2_writes_hdr_miss = 0
evict_l2_ineligible = 4096
l2_write_bytes = 0
data_size = 5965427200
c = 67601473536
l2_asize = 0
class = misc
mutex_miss = 0
snaptime = 1134.044734783
duplicate_buffers_size = 0
l2_misses = 0
mru_ghost_hits = 0
demand_metadata_hits = 29171
arc_meta_max = 100042776
c_min = 2146100480
mru_hits = 349348
demand_data_misses = 2615
duplicate_buffers = 0
demand_data_hits = 1097118
l2_free_on_write = 0
l2_feeds = 0
misses = 20068
evict_skip = 1214
evict_l2_eligible = 71168
p = 33800736768
l2_compress_successes = 0
recycle_miss = 0
l2_writes_done = 0
----------
Thu Dec 4 16:16:03 IST 2014
l2_size = 0
other_size = 18005000
c_max = 67601473536
hash_elements = 40668
duplicate_reads = 0
prefetch_data_hits = 537
l2_read_bytes = 0
demand_metadata_misses = 12540
prefetch_data_misses = 1540
hash_chain_max = 2
l2_hits = 0
l2_compress_failures = 0
mfu_ghost_hits = 0
l2_rw_clash = 0
memory_throttle_count = 0
crtime = 4.537754192
arc_meta_used = 100146232
hash_elements_max = 46675
size = 5979957304
hits = 1157458
mfu_hits = 791193
prefetch_metadata_hits = 10297
hash_chains = 85
evict_l2_cached = 0
l2_writes_sent = 0
l2_writes_error = 0
deleted = 15
hdr_size = 12560944
l2_compress_zeros = 0
l2_abort_lowmem = 0
l2_io_error = 0
l2_hdr_size = 0
l2_cksum_bad = 0
l2_evict_reading = 0
arc_meta_limit = 16900368384
prefetch_metadata_misses = 3381
l2_evict_lock_retry = 0
hash_collisions = 1162
l2_writes_hdr_miss = 0
evict_l2_ineligible = 4096
l2_write_bytes = 0
data_size = 5949391360
c = 67601473536
l2_asize = 0
class = misc
mutex_miss = 0
snaptime = 1137.04862393
duplicate_buffers_size = 0
l2_misses = 0
mru_ghost_hits = 0
demand_metadata_hits = 29187
arc_meta_max = 100145632
c_min = 2146100480
mru_hits = 355431
demand_data_misses = 2615
duplicate_buffers = 0
demand_data_hits = 1117437
l2_free_on_write = 0
l2_feeds = 0
misses = 20076
evict_skip = 1214
evict_l2_eligible = 71168
p = 33800736768
l2_compress_successes = 0
recycle_miss = 0
l2_writes_done = 0
----------
Thu Dec 4 16:16:06 IST 2014
l2_size = 0
other_size = 18052040
c_max = 67601473536
hash_elements = 45931
duplicate_reads = 0
prefetch_data_hits = 537
l2_read_bytes = 0
demand_metadata_misses = 12540
prefetch_data_misses = 1540
hash_chain_max = 2
l2_hits = 0
l2_compress_failures = 0
mfu_ghost_hits = 0
l2_rw_clash = 0
memory_throttle_count = 0
crtime = 4.537754192
arc_meta_used = 99615736
hash_elements_max = 46725
size = 6264246264
hits = 1168001
mfu_hits = 798095
prefetch_metadata_hits = 10297
hash_chains = 98
evict_l2_cached = 0
l2_writes_sent = 0
l2_writes_error = 0
deleted = 15
hdr_size = 12560944
l2_compress_zeros = 0
l2_abort_lowmem = 0
l2_io_error = 0
l2_hdr_size = 0
l2_cksum_bad = 0
l2_evict_reading = 0
arc_meta_limit = 16900368384
prefetch_metadata_misses = 3381
l2_evict_lock_retry = 0
hash_collisions = 1192
l2_writes_hdr_miss = 0
evict_l2_ineligible = 4096
l2_write_bytes = 0
data_size = 6233633280
c = 67601473536
l2_asize = 0
class = misc
mutex_miss = 0
snaptime = 1140.052771613
duplicate_buffers_size = 0
l2_misses = 0
mru_ghost_hits = 0
demand_metadata_hits = 29188
arc_meta_max = 100153872
c_min = 2146100480
mru_hits = 359072
demand_data_misses = 2615
duplicate_buffers = 0
demand_data_hits = 1127979
l2_free_on_write = 0
l2_feeds = 0
misses = 20076
evict_skip = 1214
evict_l2_eligible = 71168
p = 33800736768
l2_compress_successes = 0
recycle_miss = 0
l2_writes_done = 0
----------
Thu Dec 4 16:16:09 IST 2014
l2_size = 0
other_size = 18144240
c_max = 67601473536
hash_elements = 44399
duplicate_reads = 0
prefetch_data_hits = 537
l2_read_bytes = 0
demand_metadata_misses = 12548
prefetch_data_misses = 1540
hash_chain_max = 2
l2_hits = 0
l2_compress_failures = 0
mfu_ghost_hits = 0
l2_rw_clash = 0
memory_throttle_count = 0
crtime = 4.537754192
arc_meta_used = 99752992
hash_elements_max = 47118
size = 6250227744
hits = 1186886
mfu_hits = 811377
prefetch_metadata_hits = 10297
hash_chains = 155
evict_l2_cached = 0
l2_writes_sent = 0
l2_writes_error = 0
deleted = 15
hdr_size = 12560944
l2_compress_zeros = 0
l2_abort_lowmem = 0
l2_io_error = 0
l2_hdr_size = 0
l2_cksum_bad = 0
l2_evict_reading = 0
arc_meta_limit = 16900368384
prefetch_metadata_misses = 3381
l2_evict_lock_retry = 0
hash_collisions = 1302
l2_writes_hdr_miss = 0
evict_l2_ineligible = 4096
l2_write_bytes = 0
data_size = 6219522560
c = 67601473536
l2_asize = 0
class = misc
mutex_miss = 0
snaptime = 1143.058190189
duplicate_buffers_size = 0
l2_misses = 0
mru_ghost_hits = 0
demand_metadata_hits = 29202
arc_meta_max = 100285688
c_min = 2146100480
mru_hits = 364675
demand_data_misses = 2615
duplicate_buffers = 0
demand_data_hits = 1146850
l2_free_on_write = 0
l2_feeds = 0
misses = 20084
evict_skip = 1214
evict_l2_eligible = 71168
p = 33800736768
l2_compress_successes = 0
recycle_miss = 0
l2_writes_done = 0
----------
Thu Dec 4 16:16:12 IST 2014
l2_size = 0
other_size = 18053440
c_max = 67601473536
hash_elements = 46005
duplicate_reads = 0
prefetch_data_hits = 537
l2_read_bytes = 0
demand_metadata_misses = 12552
prefetch_data_misses = 1540
hash_chain_max = 2
l2_hits = 0
l2_compress_failures = 0
mfu_ghost_hits = 0
l2_rw_clash = 0
memory_throttle_count = 0
crtime = 4.537754192
arc_meta_used = 100254576
hash_elements_max = 47118
size = 6189538160
hits = 1194015
mfu_hits = 815523
prefetch_metadata_hits = 10297
hash_chains = 129
evict_l2_cached = 0
l2_writes_sent = 0
l2_writes_error = 0
deleted = 15
hdr_size = 12560944
l2_compress_zeros = 0
l2_abort_lowmem = 0
l2_io_error = 0
l2_hdr_size = 0
l2_cksum_bad = 0
l2_evict_reading = 0
arc_meta_limit = 16900368384
prefetch_metadata_misses = 3381
l2_evict_lock_retry = 0
hash_collisions = 1311
l2_writes_hdr_miss = 0
evict_l2_ineligible = 397312
l2_write_bytes = 0
data_size = 6158923776
c = 6190066904
l2_asize = 0
class = misc
mutex_miss = 0
snaptime = 1146.06243032
duplicate_buffers_size = 0
l2_misses = 0
mru_ghost_hits = 0
demand_metadata_hits = 29214
arc_meta_max = 100285688
c_min = 2146100480
mru_hits = 367658
demand_data_misses = 2615
duplicate_buffers = 0
demand_data_hits = 1153967
l2_free_on_write = 0
l2_feeds = 0
misses = 20088
evict_skip = 1214
evict_l2_eligible = 1011712
p = 3096114796
l2_compress_successes = 0
recycle_miss = 1
l2_writes_done = 0
----------
Thu Dec 4 16:16:15 IST 2014
l2_size = 0
other_size = 17695320
c_max = 67601473536
hash_elements = 44156
duplicate_reads = 0
prefetch_data_hits = 537
l2_read_bytes = 0
demand_metadata_misses = 12556
prefetch_data_misses = 1540
hash_chain_max = 2
l2_hits = 0
l2_compress_failures = 0
mfu_ghost_hits = 0
l2_rw_clash = 0
memory_throttle_count = 0
crtime = 4.537754192
arc_meta_used = 99370392
hash_elements_max = 47845
size = 6062509464
hits = 1214370
mfu_hits = 831061
prefetch_metadata_hits = 10297
hash_chains = 121
evict_l2_cached = 0
l2_writes_sent = 0
l2_writes_error = 0
deleted = 15
hdr_size = 12583744
l2_compress_zeros = 0
l2_abort_lowmem = 0
l2_io_error = 0
l2_hdr_size = 0
l2_cksum_bad = 0
l2_evict_reading = 0
arc_meta_limit = 16900368384
prefetch_metadata_misses = 3381
l2_evict_lock_retry = 0
hash_collisions = 1369
l2_writes_hdr_miss = 0
evict_l2_ineligible = 132255744
l2_write_bytes = 0
data_size = 6032230400
c = 6190066904
l2_asize = 0
class = misc
mutex_miss = 1
snaptime = 1149.080481297
duplicate_buffers_size = 0
l2_misses = 0
mru_ghost_hits = 0
demand_metadata_hits = 29222
arc_meta_max = 100285688
c_min = 2146100480
mru_hits = 372475
demand_data_misses = 2615
duplicate_buffers = 0
demand_data_hits = 1174314
l2_free_on_write = 0
l2_feeds = 0
misses = 20092
evict_skip = 1214
evict_l2_eligible = 285229056
p = 3401023084
l2_compress_successes = 0
recycle_miss = 219
l2_writes_done = 0
----------
Thu Dec 4 16:16:18 IST 2014
l2_size = 0
other_size = 17003120
c_max = 67601473536
hash_elements = 49196
duplicate_reads = 0
prefetch_data_hits = 537
l2_read_bytes = 0
demand_metadata_misses = 12556
prefetch_data_misses = 1540
hash_chain_max = 2
l2_hits = 0
l2_compress_failures = 0
mfu_ghost_hits = 0
l2_rw_clash = 0
memory_throttle_count = 0
crtime = 4.537754192
arc_meta_used = 97914576
hash_elements_max = 49196
size = 5597583056
hits = 1214720
mfu_hits = 831312
prefetch_metadata_hits = 10297
hash_chains = 158
evict_l2_cached = 0
l2_writes_sent = 0
l2_writes_error = 0
deleted = 15
hdr_size = 11820128
l2_compress_zeros = 0
l2_abort_lowmem = 0
l2_io_error = 0
l2_hdr_size = 0
l2_cksum_bad = 0
l2_evict_reading = 0
arc_meta_limit = 16900368384
prefetch_metadata_misses = 3381
l2_evict_lock_retry = 0
hash_collisions = 1414
l2_writes_hdr_miss = 0
evict_l2_ineligible = 132255744
l2_write_bytes = 0
data_size = 5568759808
c = 5859249032
l2_asize = 0
class = misc
mutex_miss = 1
snaptime = 1152.093743051
duplicate_buffers_size = 0
l2_misses = 0
mru_ghost_hits = 0
demand_metadata_hits = 29222
arc_meta_max = 100285688
c_min = 2146100480
mru_hits = 372574
demand_data_misses = 2615
duplicate_buffers = 0
demand_data_hits = 1174664
l2_free_on_write = 0
l2_feeds = 0
misses = 20092
evict_skip = 1214
evict_l2_eligible = 285491200
p = 3295003257
l2_compress_successes = 0
recycle_miss = 219
l2_writes_done = 0
----------
Thu Dec 4 16:16:21 IST 2014
l2_size = 0
other_size = 15860920
c_max = 67601473536
hash_elements = 44130
duplicate_reads = 0
prefetch_data_hits = 537
l2_read_bytes = 0
demand_metadata_misses = 12564
prefetch_data_misses = 1540
hash_chain_max = 2
l2_hits = 0
l2_compress_failures = 0
mfu_ghost_hits = 0
l2_rw_clash = 0
memory_throttle_count = 0
crtime = 4.537754192
arc_meta_used = 91144424
hash_elements_max = 49670
size = 4804212968
hits = 1244167
mfu_hits = 851345
prefetch_metadata_hits = 10297
hash_chains = 102
evict_l2_cached = 0
l2_writes_sent = 0
l2_writes_error = 0
deleted = 1063
hdr_size = 11332144
l2_compress_zeros = 0
l2_abort_lowmem = 0
l2_io_error = 0
l2_hdr_size = 0
l2_cksum_bad = 0
l2_evict_reading = 0
arc_meta_limit = 16900368384
prefetch_metadata_misses = 3381
l2_evict_lock_retry = 0
hash_collisions = 1433
l2_writes_hdr_miss = 0
evict_l2_ineligible = 137154560
l2_write_bytes = 0
data_size = 4777019904
c = 4879960441
l2_asize = 0
class = misc
mutex_miss = 9
snaptime = 1155.099409293
duplicate_buffers_size = 0
l2_misses = 0
mru_ghost_hits = 24
demand_metadata_hits = 29238
arc_meta_max = 100285688
c_min = 2146100480
mru_hits = 381988
demand_data_misses = 2639
duplicate_buffers = 0
demand_data_hits = 1204095
l2_free_on_write = 0
l2_feeds = 0
misses = 20124
evict_skip = 1214
evict_l2_eligible = 1330413056
p = 3274886866
l2_compress_successes = 0
recycle_miss = 519
l2_writes_done = 0
----------
Thu Dec 4 16:16:24 IST 2014
l2_size = 0
other_size = 15703800
c_max = 67601473536
hash_elements = 41125
duplicate_reads = 0
prefetch_data_hits = 537
l2_read_bytes = 0
demand_metadata_misses = 12572
prefetch_data_misses = 1540
hash_chain_max = 2
l2_hits = 0
l2_compress_failures = 0
mfu_ghost_hits = 0
l2_rw_clash = 0
memory_throttle_count = 0
crtime = 4.537754192
arc_meta_used = 85702920
hash_elements_max = 49670
size = 4396642568
hits = 1257727
mfu_hits = 860930
prefetch_metadata_hits = 10297
hash_chains = 92
evict_l2_cached = 0
l2_writes_sent = 0
l2_writes_error = 0
deleted = 3379
hdr_size = 10493456
l2_compress_zeros = 0
l2_abort_lowmem = 0
l2_io_error = 0
l2_hdr_size = 0
l2_cksum_bad = 0
l2_evict_reading = 0
arc_meta_limit = 16900368384
prefetch_metadata_misses = 3383
l2_evict_lock_retry = 0
hash_collisions = 1442
l2_writes_hdr_miss = 0
evict_l2_ineligible = 142102528
l2_write_bytes = 0
data_size = 4370445312
c = 4396690358
l2_asize = 0
class = misc
mutex_miss = 9
snaptime = 1158.10348092
duplicate_buffers_size = 0
l2_misses = 0
mru_ghost_hits = 56
demand_metadata_hits = 29252
arc_meta_max = 100285688
c_min = 2146100480
mru_hits = 385963
demand_data_misses = 2669
duplicate_buffers = 0
demand_data_hits = 1217641
l2_free_on_write = 0
l2_feeds = 0
misses = 20164
evict_skip = 1214
evict_l2_eligible = 1491629056
p = 2997814168
l2_compress_successes = 0
recycle_miss = 821
l2_writes_done = 0
----------
Thu Dec 4 16:16:27 IST 2014
l2_size = 0
other_size = 14938800
c_max = 67601473536
hash_elements = 39052
duplicate_reads = 0
prefetch_data_hits = 537
l2_read_bytes = 0
demand_metadata_misses = 12576
prefetch_data_misses = 1540
hash_chain_max = 2
l2_hits = 0
l2_compress_failures = 0
mfu_ghost_hits = 0
l2_rw_clash = 0
memory_throttle_count = 0
crtime = 4.537754192
arc_meta_used = 80821952
hash_elements_max = 49670
size = 3828283072
hits = 1264949
mfu_hits = 864148
prefetch_metadata_hits = 10297
hash_chains = 75
evict_l2_cached = 0
l2_writes_sent = 0
l2_writes_error = 0
deleted = 10138
hdr_size = 8953872
l2_compress_zeros = 0
l2_abort_lowmem = 0
l2_io_error = 0
l2_hdr_size = 0
l2_cksum_bad = 0
l2_evict_reading = 0
arc_meta_limit = 16900368384
prefetch_metadata_misses = 3383
l2_evict_lock_retry = 0
hash_collisions = 1454
l2_writes_hdr_miss = 0
evict_l2_ineligible = 144691200
l2_write_bytes = 0
data_size = 3804390400
c = 3830309700
l2_asize = 0
class = misc
mutex_miss = 12
snaptime = 1161.109480472
duplicate_buffers_size = 0
l2_misses = 0
mru_ghost_hits = 77
demand_metadata_hits = 29256
arc_meta_max = 100285688
c_min = 2146100480
mru_hits = 389967
demand_data_misses = 2692
duplicate_buffers = 0
demand_data_hits = 1224859
l2_free_on_write = 0
l2_feeds = 0
misses = 20191
evict_skip = 1214
evict_l2_eligible = 1951429632
p = 2764099257
l2_compress_successes = 0
recycle_miss = 972
l2_writes_done = 0
----------