[OmniOS-discuss] Kernel stability.

Kobi Schwartz orilali at evogene.com
Fri Dec 5 06:56:34 UTC 2014


Hi,

My name is Kobi and I work with a company called Evogene.

We have a storage server based on a Supermicro X9DRD-7LN4F (Intel(R) 
Xeon(R) CPU E5-2609, 64GB RAM) equipped with Intel X520-DA2 10Gb 
Ethernet card.

Two weeks ago we encountered a problem which regards Omnios (r15012) and 
OpenIndiana (151_a8) kernel stability.

The system is a clean install of r151012, only configuration is 
networking and zpool (raidz3 1 vdev of 8 disks of data and 3 parity).

The storage has a single NFS share mounted (nfsvers=3) on an 8 grid 
nodes (running CentOS6.6) with the following mount options :

async,tcp,vers=3,noacl,timeo=20,rw,bg,hard,intr,rsize=65536,wsize=65536 
0 0.

Under normal conditions (random reads and writes) we get great 
performance and stability but if we increase nfs usage the Kernel memory 
usage increases until the machine eventually halts.

To test it we used on each node iozone ( Version $Revision: 3.394 ) 
which simultaneously write and reads 12 files with a size of 500MB.
(iozone -r 64k -i 0 -i 1 -s 500m -t 12 -F 
/evolon2/iozone/`hostname`.{1..12}).

While the iozone's are running the memory of Kernel (as seen by echo 
'::memstat'| mdb -k) is steadily increasing until the machine is dead.

For clarification, when the machine is dead/halts:
There is no crash files.
There is no new messages in /var/adm/messages.
It doesn't respond to pings (and obviously other network services).
Waiting doesn't help.
It can only be brought to life by power cycling it.

I attached 3 files: memstat, top and arc-kstat which I hope will assist 
to understand why the storage died/halted

Thank you in advance,

Kobi.


memstat

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://omniosce.org/ml-archive/attachments/20141205/761856c7/attachment-0001.html>
-------------- next part --------------
Thu Dec  4 16:14:39 IST 2014

l2_size                   = 0                   
other_size                = 16487240            
c_max                     = 67601473536         
hash_elements             = 36191               
duplicate_reads           = 0                   
prefetch_data_hits        = 537                 
l2_read_bytes             = 0                   
demand_metadata_misses    = 12438               
prefetch_data_misses      = 1540                
hash_chain_max            = 2                   
l2_hits                   = 0                   
l2_compress_failures      = 0                   
mfu_ghost_hits            = 0                   
l2_rw_clash               = 0                   
memory_throttle_count     = 0                   
crtime                    = 4.537754192         
arc_meta_used             = 97172664            
hash_elements_max         = 41359               
size                      = 5312841912          
hits                      = 750510              
mfu_hits                  = 507924              
prefetch_metadata_hits    = 10297               
hash_chains               = 54                  
evict_l2_cached           = 0                   
l2_writes_sent            = 0                   
l2_writes_error           = 0                   
deleted                   = 15                  
hdr_size                  = 12559216            
l2_compress_zeros         = 0                   
l2_abort_lowmem           = 0                   
l2_io_error               = 0                   
l2_hdr_size               = 0                   
l2_cksum_bad              = 0                   
l2_evict_reading          = 0                   
arc_meta_limit            = 16900368384         
prefetch_metadata_misses  = 3381                
l2_evict_lock_retry       = 0                   
hash_collisions           = 519                 
l2_writes_hdr_miss        = 0                   
evict_l2_ineligible       = 4096                
l2_write_bytes            = 0                   
data_size                 = 5283795456          
c                         = 67601473536         
l2_asize                  = 0                   
class                     = misc                
mutex_miss                = 0                   
snaptime                  = 1052.750004426      
duplicate_buffers_size    = 0                   
l2_misses                 = 0                   
mru_ghost_hits            = 0                   
demand_metadata_hits      = 28989               
arc_meta_max              = 97686696            
c_min                     = 2146100480          
mru_hits                  = 231752              
demand_data_misses        = 2615                
duplicate_buffers         = 0                   
demand_data_hits          = 710687              
l2_free_on_write          = 0                   
l2_feeds                  = 0                   
misses                    = 19974               
evict_skip                = 1214                
evict_l2_eligible         = 71168               
p                         = 33800736768         
l2_compress_successes     = 0                   
recycle_miss              = 0                   
l2_writes_done            = 0                   
----------

Thu Dec  4 16:14:42 IST 2014

l2_size                   = 0                   
other_size                = 16557200            
c_max                     = 67601473536         
hash_elements             = 35929               
duplicate_reads           = 0                   
prefetch_data_hits        = 537                 
l2_read_bytes             = 0                   
demand_metadata_misses    = 12446               
prefetch_data_misses      = 1540                
hash_chain_max            = 2                   
l2_hits                   = 0                   
l2_compress_failures      = 0                   
mfu_ghost_hits            = 0                   
l2_rw_clash               = 0                   
memory_throttle_count     = 0                   
crtime                    = 4.537754192         
arc_meta_used             = 97848832            
hash_elements_max         = 41359               
size                      = 5242345984          
hits                      = 765628              
mfu_hits                  = 519125              
prefetch_metadata_hits    = 10297               
hash_chains               = 53                  
evict_l2_cached           = 0                   
l2_writes_sent            = 0                   
l2_writes_error           = 0                   
deleted                   = 15                  
hdr_size                  = 12559216            
l2_compress_zeros         = 0                   
l2_abort_lowmem           = 0                   
l2_io_error               = 0                   
l2_hdr_size               = 0                   
l2_cksum_bad              = 0                   
l2_evict_reading          = 0                   
arc_meta_limit            = 16900368384         
prefetch_metadata_misses  = 3381                
l2_evict_lock_retry       = 0                   
hash_collisions           = 544                 
l2_writes_hdr_miss        = 0                   
evict_l2_ineligible       = 4096                
l2_write_bytes            = 0                   
data_size                 = 5213229568          
c                         = 67601473536         
l2_asize                  = 0                   
class                     = misc                
mutex_miss                = 0                   
snaptime                  = 1055.756679096      
duplicate_buffers_size    = 0                   
l2_misses                 = 0                   
mru_ghost_hits            = 0                   
demand_metadata_hits      = 28999               
arc_meta_max              = 97815864            
c_min                     = 2146100480          
mru_hits                  = 235669              
demand_data_misses        = 2615                
duplicate_buffers         = 0                   
demand_data_hits          = 725795              
l2_free_on_write          = 0                   
l2_feeds                  = 0                   
misses                    = 19982               
evict_skip                = 1214                
evict_l2_eligible         = 71168               
p                         = 33800736768         
l2_compress_successes     = 0                   
recycle_miss              = 0                   
l2_writes_done            = 0                   
----------

Thu Dec  4 16:14:45 IST 2014

l2_size                   = 0                   
other_size                = 16815440            
c_max                     = 67601473536         
hash_elements             = 34070               
duplicate_reads           = 0                   
prefetch_data_hits        = 537                 
l2_read_bytes             = 0                   
demand_metadata_misses    = 12446               
prefetch_data_misses      = 1540                
hash_chain_max            = 2                   
l2_hits                   = 0                   
l2_compress_failures      = 0                   
mfu_ghost_hits            = 0                   
l2_rw_clash               = 0                   
memory_throttle_count     = 0                   
crtime                    = 4.537754192         
arc_meta_used             = 97619648            
hash_elements_max         = 41359               
size                      = 5487483584          
hits                      = 791669              
mfu_hits                  = 537122              
prefetch_metadata_hits    = 10297               
hash_chains               = 50                  
evict_l2_cached           = 0                   
l2_writes_sent            = 0                   
l2_writes_error           = 0                   
deleted                   = 15                  
hdr_size                  = 12559216            
l2_compress_zeros         = 0                   
l2_abort_lowmem           = 0                   
l2_io_error               = 0                   
l2_hdr_size               = 0                   
l2_cksum_bad              = 0                   
l2_evict_reading          = 0                   
arc_meta_limit            = 16900368384         
prefetch_metadata_misses  = 3381                
l2_evict_lock_retry       = 0                   
hash_collisions           = 581                 
l2_writes_hdr_miss        = 0                   
evict_l2_ineligible       = 4096                
l2_write_bytes            = 0                   
data_size                 = 5458108928          
c                         = 67601473536         
l2_asize                  = 0                   
class                     = misc                
mutex_miss                = 0                   
snaptime                  = 1058.767796546      
duplicate_buffers_size    = 0                   
l2_misses                 = 0                   
mru_ghost_hits            = 0                   
demand_metadata_hits      = 29003               
arc_meta_max              = 97968040            
c_min                     = 2146100480          
mru_hits                  = 243713              
demand_data_misses        = 2615                
duplicate_buffers         = 0                   
demand_data_hits          = 751832              
l2_free_on_write          = 0                   
l2_feeds                  = 0                   
misses                    = 19982               
evict_skip                = 1214                
evict_l2_eligible         = 71168               
p                         = 33800736768         
l2_compress_successes     = 0                   
recycle_miss              = 0                   
l2_writes_done            = 0                   
----------

Thu Dec  4 16:14:48 IST 2014

l2_size                   = 0                   
other_size                = 16858360            
c_max                     = 67601473536         
hash_elements             = 36794               
duplicate_reads           = 0                   
prefetch_data_hits        = 537                 
l2_read_bytes             = 0                   
demand_metadata_misses    = 12454               
prefetch_data_misses      = 1540                
hash_chain_max            = 2                   
l2_hits                   = 0                   
l2_compress_failures      = 0                   
mfu_ghost_hits            = 0                   
l2_rw_clash               = 0                   
memory_throttle_count     = 0                   
crtime                    = 4.537754192         
arc_meta_used             = 98313832            
hash_elements_max         = 41359               
size                      = 5462880872          
hits                      = 802717              
mfu_hits                  = 544723              
prefetch_metadata_hits    = 10297               
hash_chains               = 75                  
evict_l2_cached           = 0                   
l2_writes_sent            = 0                   
l2_writes_error           = 0                   
deleted                   = 15                  
hdr_size                  = 12559216            
l2_compress_zeros         = 0                   
l2_abort_lowmem           = 0                   
l2_io_error               = 0                   
l2_hdr_size               = 0                   
l2_cksum_bad              = 0                   
l2_evict_reading          = 0                   
arc_meta_limit            = 16900368384         
prefetch_metadata_misses  = 3381                
l2_evict_lock_retry       = 0                   
hash_collisions           = 626                 
l2_writes_hdr_miss        = 0                   
evict_l2_ineligible       = 4096                
l2_write_bytes            = 0                   
data_size                 = 5433463296          
c                         = 67601473536         
l2_asize                  = 0                   
class                     = misc                
mutex_miss                = 0                   
snaptime                  = 1061.787519516      
duplicate_buffers_size    = 0                   
l2_misses                 = 0                   
mru_ghost_hits            = 0                   
demand_metadata_hits      = 29022               
arc_meta_max              = 98312872            
c_min                     = 2146100480          
mru_hits                  = 247160              
demand_data_misses        = 2615                
duplicate_buffers         = 0                   
demand_data_hits          = 762861              
l2_free_on_write          = 0                   
l2_feeds                  = 0                   
misses                    = 19990               
evict_skip                = 1214                
evict_l2_eligible         = 71168               
p                         = 33800736768         
l2_compress_successes     = 0                   
recycle_miss              = 0                   
l2_writes_done            = 0                   
----------

Thu Dec  4 16:14:51 IST 2014

l2_size                   = 0                   
other_size                = 16888240            
c_max                     = 67601473536         
hash_elements             = 40590               
duplicate_reads           = 0                   
prefetch_data_hits        = 537                 
l2_read_bytes             = 0                   
demand_metadata_misses    = 12458               
prefetch_data_misses      = 1540                
hash_chain_max            = 2                   
l2_hits                   = 0                   
l2_compress_failures      = 0                   
mfu_ghost_hits            = 0                   
l2_rw_clash               = 0                   
memory_throttle_count     = 0                   
crtime                    = 4.537754192         
arc_meta_used             = 98425632            
hash_elements_max         = 42641               
size                      = 5669168928          
hits                      = 815255              
mfu_hits                  = 552397              
prefetch_metadata_hits    = 10297               
hash_chains               = 86                  
evict_l2_cached           = 0                   
l2_writes_sent            = 0                   
l2_writes_error           = 0                   
deleted                   = 15                  
hdr_size                  = 12559216            
l2_compress_zeros         = 0                   
l2_abort_lowmem           = 0                   
l2_io_error               = 0                   
l2_hdr_size               = 0                   
l2_cksum_bad              = 0                   
l2_evict_reading          = 0                   
arc_meta_limit            = 16900368384         
prefetch_metadata_misses  = 3381                
l2_evict_lock_retry       = 0                   
hash_collisions           = 664                 
l2_writes_hdr_miss        = 0                   
evict_l2_ineligible       = 4096                
l2_write_bytes            = 0                   
data_size                 = 5639721472          
c                         = 67601473536         
l2_asize                  = 0                   
class                     = misc                
mutex_miss                = 0                   
snaptime                  = 1064.791199302      
duplicate_buffers_size    = 0                   
l2_misses                 = 0                   
mru_ghost_hits            = 0                   
demand_metadata_hits      = 29029               
arc_meta_max              = 98388464            
c_min                     = 2146100480          
mru_hits                  = 252024              
demand_data_misses        = 2615                
duplicate_buffers         = 0                   
demand_data_hits          = 775392              
l2_free_on_write          = 0                   
l2_feeds                  = 0                   
misses                    = 19994               
evict_skip                = 1214                
evict_l2_eligible         = 71168               
p                         = 33800736768         
l2_compress_successes     = 0                   
recycle_miss              = 0                   
l2_writes_done            = 0                   
----------

Thu Dec  4 16:14:54 IST 2014

l2_size                   = 0                   
other_size                = 17262640            
c_max                     = 67601473536         
hash_elements             = 36737               
duplicate_reads           = 0                   
prefetch_data_hits        = 537                 
l2_read_bytes             = 0                   
demand_metadata_misses    = 12462               
prefetch_data_misses      = 1540                
hash_chain_max            = 2                   
l2_hits                   = 0                   
l2_compress_failures      = 0                   
mfu_ghost_hits            = 0                   
l2_rw_clash               = 0                   
memory_throttle_count     = 0                   
crtime                    = 4.537754192         
arc_meta_used             = 98304416            
hash_elements_max         = 42641               
size                      = 6396890528          
hits                      = 840144              
mfu_hits                  = 570407              
prefetch_metadata_hits    = 10297               
hash_chains               = 80                  
evict_l2_cached           = 0                   
l2_writes_sent            = 0                   
l2_writes_error           = 0                   
deleted                   = 15                  
hdr_size                  = 12559216            
l2_compress_zeros         = 0                   
l2_abort_lowmem           = 0                   
l2_io_error               = 0                   
l2_hdr_size               = 0                   
l2_cksum_bad              = 0                   
l2_evict_reading          = 0                   
arc_meta_limit            = 16900368384         
prefetch_metadata_misses  = 3381                
l2_evict_lock_retry       = 0                   
hash_collisions           = 690                 
l2_writes_hdr_miss        = 0                   
evict_l2_ineligible       = 4096                
l2_write_bytes            = 0                   
data_size                 = 6367068672          
c                         = 67601473536         
l2_asize                  = 0                   
class                     = misc                
mutex_miss                = 0                   
snaptime                  = 1067.797220708      
duplicate_buffers_size    = 0                   
l2_misses                 = 0                   
mru_ghost_hits            = 0                   
demand_metadata_hits      = 29038               
arc_meta_max              = 98730168            
c_min                     = 2146100480          
mru_hits                  = 258903              
demand_data_misses        = 2615                
duplicate_buffers         = 0                   
demand_data_hits          = 800272              
l2_free_on_write          = 0                   
l2_feeds                  = 0                   
misses                    = 19998               
evict_skip                = 1214                
evict_l2_eligible         = 71168               
p                         = 33800736768         
l2_compress_successes     = 0                   
recycle_miss              = 0                   
l2_writes_done            = 0                   
----------

Thu Dec  4 16:14:57 IST 2014

l2_size                   = 0                   
other_size                = 17330840            
c_max                     = 67601473536         
hash_elements             = 39702               
duplicate_reads           = 0                   
prefetch_data_hits        = 537                 
l2_read_bytes             = 0                   
demand_metadata_misses    = 12466               
prefetch_data_misses      = 1540                
hash_chain_max            = 2                   
l2_hits                   = 0                   
l2_compress_failures      = 0                   
mfu_ghost_hits            = 0                   
l2_rw_clash               = 0                   
memory_throttle_count     = 0                   
crtime                    = 4.537754192         
arc_meta_used             = 99036168            
hash_elements_max         = 42641               
size                      = 6258292744          
hits                      = 851791              
mfu_hits                  = 578190              
prefetch_metadata_hits    = 10297               
hash_chains               = 63                  
evict_l2_cached           = 0                   
l2_writes_sent            = 0                   
l2_writes_error           = 0                   
deleted                   = 15                  
hdr_size                  = 12559216            
l2_compress_zeros         = 0                   
l2_abort_lowmem           = 0                   
l2_io_error               = 0                   
l2_hdr_size               = 0                   
l2_cksum_bad              = 0                   
l2_evict_reading          = 0                   
arc_meta_limit            = 16900368384         
prefetch_metadata_misses  = 3381                
l2_evict_lock_retry       = 0                   
hash_collisions           = 710                 
l2_writes_hdr_miss        = 0                   
evict_l2_ineligible       = 4096                
l2_write_bytes            = 0                   
data_size                 = 6228402688          
c                         = 67601473536         
l2_asize                  = 0                   
class                     = misc                
mutex_miss                = 0                   
snaptime                  = 1070.806236777      
duplicate_buffers_size    = 0                   
l2_misses                 = 0                   
mru_ghost_hits            = 0                   
demand_metadata_hits      = 29044               
arc_meta_max              = 99036168            
c_min                     = 2146100480          
mru_hits                  = 262767              
demand_data_misses        = 2615                
duplicate_buffers         = 0                   
demand_data_hits          = 811913              
l2_free_on_write          = 0                   
l2_feeds                  = 0                   
misses                    = 20002               
evict_skip                = 1214                
evict_l2_eligible         = 71168               
p                         = 33800736768         
l2_compress_successes     = 0                   
recycle_miss              = 0                   
l2_writes_done            = 0                   
----------

Thu Dec  4 16:15:00 IST 2014

l2_size                   = 0                   
other_size                = 17276800            
c_max                     = 67601473536         
hash_elements             = 39938               
duplicate_reads           = 0                   
prefetch_data_hits        = 537                 
l2_read_bytes             = 0                   
demand_metadata_misses    = 12470               
prefetch_data_misses      = 1540                
hash_chain_max            = 2                   
l2_hits                   = 0                   
l2_compress_failures      = 0                   
mfu_ghost_hits            = 0                   
l2_rw_clash               = 0                   
memory_throttle_count     = 0                   
crtime                    = 4.537754192         
arc_meta_used             = 98994416            
hash_elements_max         = 42641               
size                      = 5947348208          
hits                      = 854591              
mfu_hits                  = 580209              
prefetch_metadata_hits    = 10297               
hash_chains               = 64                  
evict_l2_cached           = 0                   
l2_writes_sent            = 0                   
l2_writes_error           = 0                   
deleted                   = 15                  
hdr_size                  = 12559216            
l2_compress_zeros         = 0                   
l2_abort_lowmem           = 0                   
l2_io_error               = 0                   
l2_hdr_size               = 0                   
l2_cksum_bad              = 0                   
l2_evict_reading          = 0                   
arc_meta_limit            = 16900368384         
prefetch_metadata_misses  = 3381                
l2_evict_lock_retry       = 0                   
hash_collisions           = 713                 
l2_writes_hdr_miss        = 0                   
evict_l2_ineligible       = 4096                
l2_write_bytes            = 0                   
data_size                 = 5917512192          
c                         = 67601473536         
l2_asize                  = 0                   
class                     = misc                
mutex_miss                = 0                   
snaptime                  = 1073.811610852      
duplicate_buffers_size    = 0                   
l2_misses                 = 0                   
mru_ghost_hits            = 0                   
demand_metadata_hits      = 29051               
arc_meta_max              = 99051168            
c_min                     = 2146100480          
mru_hits                  = 263548              
demand_data_misses        = 2615                
duplicate_buffers         = 0                   
demand_data_hits          = 814706              
l2_free_on_write          = 0                   
l2_feeds                  = 0                   
misses                    = 20006               
evict_skip                = 1214                
evict_l2_eligible         = 71168               
p                         = 33800736768         
l2_compress_successes     = 0                   
recycle_miss              = 0                   
l2_writes_done            = 0                   
----------

Thu Dec  4 16:15:03 IST 2014

l2_size                   = 0                   
other_size                = 17471000            
c_max                     = 67601473536         
hash_elements             = 38061               
duplicate_reads           = 0                   
prefetch_data_hits        = 537                 
l2_read_bytes             = 0                   
demand_metadata_misses    = 12470               
prefetch_data_misses      = 1540                
hash_chain_max            = 2                   
l2_hits                   = 0                   
l2_compress_failures      = 0                   
mfu_ghost_hits            = 0                   
l2_rw_clash               = 0                   
memory_throttle_count     = 0                   
crtime                    = 4.537754192         
arc_meta_used             = 98656136            
hash_elements_max         = 42641               
size                      = 6162492296          
hits                      = 881460              
mfu_hits                  = 598502              
prefetch_metadata_hits    = 10297               
hash_chains               = 77                  
evict_l2_cached           = 0                   
l2_writes_sent            = 0                   
l2_writes_error           = 0                   
deleted                   = 15                  
hdr_size                  = 12559216            
l2_compress_zeros         = 0                   
l2_abort_lowmem           = 0                   
l2_io_error               = 0                   
l2_hdr_size               = 0                   
l2_cksum_bad              = 0                   
l2_evict_reading          = 0                   
arc_meta_limit            = 16900368384         
prefetch_metadata_misses  = 3381                
l2_evict_lock_retry       = 0                   
hash_collisions           = 756                 
l2_writes_hdr_miss        = 0                   
evict_l2_ineligible       = 4096                
l2_write_bytes            = 0                   
data_size                 = 6132462080          
c                         = 67601473536         
l2_asize                  = 0                   
class                     = misc                
mutex_miss                = 0                   
snaptime                  = 1076.82642884       
duplicate_buffers_size    = 0                   
l2_misses                 = 0                   
mru_ghost_hits            = 0                   
demand_metadata_hits      = 29052               
arc_meta_max              = 99051168            
c_min                     = 2146100480          
mru_hits                  = 272124              
demand_data_misses        = 2615                
duplicate_buffers         = 0                   
demand_data_hits          = 841574              
l2_free_on_write          = 0                   
l2_feeds                  = 0                   
misses                    = 20006               
evict_skip                = 1214                
evict_l2_eligible         = 71168               
p                         = 33800736768         
l2_compress_successes     = 0                   
recycle_miss              = 0                   
l2_writes_done            = 0                   
----------

Thu Dec  4 16:15:06 IST 2014

l2_size                   = 0                   
other_size                = 17531840            
c_max                     = 67601473536         
hash_elements             = 38754               
duplicate_reads           = 0                   
prefetch_data_hits        = 537                 
l2_read_bytes             = 0                   
demand_metadata_misses    = 12478               
prefetch_data_misses      = 1540                
hash_chain_max            = 2                   
l2_hits                   = 0                   
l2_compress_failures      = 0                   
mfu_ghost_hits            = 0                   
l2_rw_clash               = 0                   
memory_throttle_count     = 0                   
crtime                    = 4.537754192         
arc_meta_used             = 98766128            
hash_elements_max         = 42641               
size                      = 6131800368          
hits                      = 891251              
mfu_hits                  = 605449              
prefetch_metadata_hits    = 10297               
hash_chains               = 83                  
evict_l2_cached           = 0                   
l2_writes_sent            = 0                   
l2_writes_error           = 0                   
deleted                   = 15                  
hdr_size                  = 12559216            
l2_compress_zeros         = 0                   
l2_abort_lowmem           = 0                   
l2_io_error               = 0                   
l2_hdr_size               = 0                   
l2_cksum_bad              = 0                   
l2_evict_reading          = 0                   
arc_meta_limit            = 16900368384         
prefetch_metadata_misses  = 3381                
l2_evict_lock_retry       = 0                   
hash_collisions           = 774                 
l2_writes_hdr_miss        = 0                   
evict_l2_ineligible       = 4096                
l2_write_bytes            = 0                   
data_size                 = 6101709312          
c                         = 67601473536         
l2_asize                  = 0                   
class                     = misc                
mutex_miss                = 0                   
snaptime                  = 1079.87068534       
duplicate_buffers_size    = 0                   
l2_misses                 = 0                   
mru_ghost_hits            = 0                   
demand_metadata_hits      = 29066               
arc_meta_max              = 99360176            
c_min                     = 2146100480          
mru_hits                  = 274968              
demand_data_misses        = 2615                
duplicate_buffers         = 0                   
demand_data_hits          = 851351              
l2_free_on_write          = 0                   
l2_feeds                  = 0                   
misses                    = 20014               
evict_skip                = 1214                
evict_l2_eligible         = 71168               
p                         = 33800736768         
l2_compress_successes     = 0                   
recycle_miss              = 0                   
l2_writes_done            = 0                   
----------

Thu Dec  4 16:15:09 IST 2014

l2_size                   = 0                   
other_size                = 17593040            
c_max                     = 67601473536         
hash_elements             = 40255               
duplicate_reads           = 0                   
prefetch_data_hits        = 537                 
l2_read_bytes             = 0                   
demand_metadata_misses    = 12478               
prefetch_data_misses      = 1540                
hash_chain_max            = 2                   
l2_hits                   = 0                   
l2_compress_failures      = 0                   
mfu_ghost_hits            = 0                   
l2_rw_clash               = 0                   
memory_throttle_count     = 0                   
crtime                    = 4.537754192         
arc_meta_used             = 98860096            
hash_elements_max         = 42641               
size                      = 6205425728          
hits                      = 902579              
mfu_hits                  = 612627              
prefetch_metadata_hits    = 10297               
hash_chains               = 81                  
evict_l2_cached           = 0                   
l2_writes_sent            = 0                   
l2_writes_error           = 0                   
deleted                   = 15                  
hdr_size                  = 12559216            
l2_compress_zeros         = 0                   
l2_abort_lowmem           = 0                   
l2_io_error               = 0                   
l2_hdr_size               = 0                   
l2_cksum_bad              = 0                   
l2_evict_reading          = 0                   
arc_meta_limit            = 16900368384         
prefetch_metadata_misses  = 3381                
l2_evict_lock_retry       = 0                   
hash_collisions           = 794                 
l2_writes_hdr_miss        = 0                   
evict_l2_ineligible       = 4096                
l2_write_bytes            = 0                   
data_size                 = 6175273472          
c                         = 67601473536         
l2_asize                  = 0                   
class                     = misc                
mutex_miss                = 0                   
snaptime                  = 1082.875826687      
duplicate_buffers_size    = 0                   
l2_misses                 = 0                   
mru_ghost_hits            = 0                   
demand_metadata_hits      = 29066               
arc_meta_max              = 99360176            
c_min                     = 2146100480          
mru_hits                  = 279118              
demand_data_misses        = 2615                
duplicate_buffers         = 0                   
demand_data_hits          = 862679              
l2_free_on_write          = 0                   
l2_feeds                  = 0                   
misses                    = 20014               
evict_skip                = 1214                
evict_l2_eligible         = 71168               
p                         = 33800736768         
l2_compress_successes     = 0                   
recycle_miss              = 0                   
l2_writes_done            = 0                   
----------

Thu Dec  4 16:15:12 IST 2014

l2_size                   = 0                   
other_size                = 17598240            
c_max                     = 67601473536         
hash_elements             = 43117               
duplicate_reads           = 0                   
prefetch_data_hits        = 537                 
l2_read_bytes             = 0                   
demand_metadata_misses    = 12478               
prefetch_data_misses      = 1540                
hash_chain_max            = 2                   
l2_hits                   = 0                   
l2_compress_failures      = 0                   
mfu_ghost_hits            = 0                   
l2_rw_clash               = 0                   
memory_throttle_count     = 0                   
crtime                    = 4.537754192         
arc_meta_used             = 98865296            
hash_elements_max         = 43118               
size                      = 6013541520          
hits                      = 905362              
mfu_hits                  = 613511              
prefetch_metadata_hits    = 10297               
hash_chains               = 111                 
evict_l2_cached           = 0                   
l2_writes_sent            = 0                   
l2_writes_error           = 0                   
deleted                   = 15                  
hdr_size                  = 12559216            
l2_compress_zeros         = 0                   
l2_abort_lowmem           = 0                   
l2_io_error               = 0                   
l2_hdr_size               = 0                   
l2_cksum_bad              = 0                   
l2_evict_reading          = 0                   
arc_meta_limit            = 16900368384         
prefetch_metadata_misses  = 3381                
l2_evict_lock_retry       = 0                   
hash_collisions           = 831                 
l2_writes_hdr_miss        = 0                   
evict_l2_ineligible       = 4096                
l2_write_bytes            = 0                   
data_size                 = 5983384064          
c                         = 67601473536         
l2_asize                  = 0                   
class                     = misc                
mutex_miss                = 0                   
snaptime                  = 1085.879514131      
duplicate_buffers_size    = 0                   
l2_misses                 = 0                   
mru_ghost_hits            = 0                   
demand_metadata_hits      = 29066               
arc_meta_max              = 99360176            
c_min                     = 2146100480          
mru_hits                  = 281017              
demand_data_misses        = 2615                
duplicate_buffers         = 0                   
demand_data_hits          = 865462              
l2_free_on_write          = 0                   
l2_feeds                  = 0                   
misses                    = 20014               
evict_skip                = 1214                
evict_l2_eligible         = 71168               
p                         = 33800736768         
l2_compress_successes     = 0                   
recycle_miss              = 0                   
l2_writes_done            = 0                   
----------

Thu Dec  4 16:15:15 IST 2014

l2_size                   = 0                   
other_size                = 17603440            
c_max                     = 67601473536         
hash_elements             = 43174               
duplicate_reads           = 0                   
prefetch_data_hits        = 537                 
l2_read_bytes             = 0                   
demand_metadata_misses    = 12482               
prefetch_data_misses      = 1540                
hash_chain_max            = 2                   
l2_hits                   = 0                   
l2_compress_failures      = 0                   
mfu_ghost_hits            = 0                   
l2_rw_clash               = 0                   
memory_throttle_count     = 0                   
crtime                    = 4.537754192         
arc_meta_used             = 99480800            
hash_elements_max         = 43929               
size                      = 5807325408          
hits                      = 908319              
mfu_hits                  = 615797              
prefetch_metadata_hits    = 10297               
hash_chains               = 102                 
evict_l2_cached           = 0                   
l2_writes_sent            = 0                   
l2_writes_error           = 0                   
deleted                   = 15                  
hdr_size                  = 12559216            
l2_compress_zeros         = 0                   
l2_abort_lowmem           = 0                   
l2_io_error               = 0                   
l2_hdr_size               = 0                   
l2_cksum_bad              = 0                   
l2_evict_reading          = 0                   
arc_meta_limit            = 16900368384         
prefetch_metadata_misses  = 3381                
l2_evict_lock_retry       = 0                   
hash_collisions           = 832                 
l2_writes_hdr_miss        = 0                   
evict_l2_ineligible       = 4096                
l2_write_bytes            = 0                   
data_size                 = 5777162752          
c                         = 67601473536         
l2_asize                  = 0                   
class                     = misc                
mutex_miss                = 0                   
snaptime                  = 1088.884358936      
duplicate_buffers_size    = 0                   
l2_misses                 = 0                   
mru_ghost_hits            = 0                   
demand_metadata_hits      = 29075               
arc_meta_max              = 99480800            
c_min                     = 2146100480          
mru_hits                  = 281688              
demand_data_misses        = 2615                
duplicate_buffers         = 0                   
demand_data_hits          = 868410              
l2_free_on_write          = 0                   
l2_feeds                  = 0                   
misses                    = 20018               
evict_skip                = 1214                
evict_l2_eligible         = 71168               
p                         = 33800736768         
l2_compress_successes     = 0                   
recycle_miss              = 0                   
l2_writes_done            = 0                   
----------

Thu Dec  4 16:15:18 IST 2014

l2_size                   = 0                   
other_size                = 17668200            
c_max                     = 67601473536         
hash_elements             = 41509               
duplicate_reads           = 0                   
prefetch_data_hits        = 537                 
l2_read_bytes             = 0                   
demand_metadata_misses    = 12488               
prefetch_data_misses      = 1540                
hash_chain_max            = 2                   
l2_hits                   = 0                   
l2_compress_failures      = 0                   
mfu_ghost_hits            = 0                   
l2_rw_clash               = 0                   
memory_throttle_count     = 0                   
crtime                    = 4.537754192         
arc_meta_used             = 98984408            
hash_elements_max         = 43929               
size                      = 5884816856          
hits                      = 927065              
mfu_hits                  = 630052              
prefetch_metadata_hits    = 10297               
hash_chains               = 93                  
evict_l2_cached           = 0                   
l2_writes_sent            = 0                   
l2_writes_error           = 0                   
deleted                   = 15                  
hdr_size                  = 12559216            
l2_compress_zeros         = 0                   
l2_abort_lowmem           = 0                   
l2_io_error               = 0                   
l2_hdr_size               = 0                   
l2_cksum_bad              = 0                   
l2_evict_reading          = 0                   
arc_meta_limit            = 16900368384         
prefetch_metadata_misses  = 3381                
l2_evict_lock_retry       = 0                   
hash_collisions           = 843                 
l2_writes_hdr_miss        = 0                   
evict_l2_ineligible       = 4096                
l2_write_bytes            = 0                   
data_size                 = 5854589440          
c                         = 67601473536         
l2_asize                  = 0                   
class                     = misc                
mutex_miss                = 0                   
snaptime                  = 1091.888514904      
duplicate_buffers_size    = 0                   
l2_misses                 = 0                   
mru_ghost_hits            = 0                   
demand_metadata_hits      = 29087               
arc_meta_max              = 99568328            
c_min                     = 2146100480          
mru_hits                  = 286179              
demand_data_misses        = 2615                
duplicate_buffers         = 0                   
demand_data_hits          = 887144              
l2_free_on_write          = 0                   
l2_feeds                  = 0                   
misses                    = 20024               
evict_skip                = 1214                
evict_l2_eligible         = 71168               
p                         = 33800736768         
l2_compress_successes     = 0                   
recycle_miss              = 0                   
l2_writes_done            = 0                   
----------

Thu Dec  4 16:15:21 IST 2014

l2_size                   = 0                   
other_size                = 17646400            
c_max                     = 67601473536         
hash_elements             = 39031               
duplicate_reads           = 0                   
prefetch_data_hits        = 537                 
l2_read_bytes             = 0                   
demand_metadata_misses    = 12488               
prefetch_data_misses      = 1540                
hash_chain_max            = 2                   
l2_hits                   = 0                   
l2_compress_failures      = 0                   
mfu_ghost_hits            = 0                   
l2_rw_clash               = 0                   
memory_throttle_count     = 0                   
crtime                    = 4.537754192         
arc_meta_used             = 98978992            
hash_elements_max         = 43929               
size                      = 5865412784          
hits                      = 955131              
mfu_hits                  = 649821              
prefetch_metadata_hits    = 10297               
hash_chains               = 76                  
evict_l2_cached           = 0                   
l2_writes_sent            = 0                   
l2_writes_error           = 0                   
deleted                   = 15                  
hdr_size                  = 12559216            
l2_compress_zeros         = 0                   
l2_abort_lowmem           = 0                   
l2_io_error               = 0                   
l2_hdr_size               = 0                   
l2_cksum_bad              = 0                   
l2_evict_reading          = 0                   
arc_meta_limit            = 16900368384         
prefetch_metadata_misses  = 3381                
l2_evict_lock_retry       = 0                   
hash_collisions           = 857                 
l2_writes_hdr_miss        = 0                   
evict_l2_ineligible       = 4096                
l2_write_bytes            = 0                   
data_size                 = 5835207168          
c                         = 67601473536         
l2_asize                  = 0                   
class                     = misc                
mutex_miss                = 0                   
snaptime                  = 1094.945818674      
duplicate_buffers_size    = 0                   
l2_misses                 = 0                   
mru_ghost_hits            = 0                   
demand_metadata_hits      = 29087               
arc_meta_max              = 99568328            
c_min                     = 2146100480          
mru_hits                  = 294476              
demand_data_misses        = 2615                
duplicate_buffers         = 0                   
demand_data_hits          = 915210              
l2_free_on_write          = 0                   
l2_feeds                  = 0                   
misses                    = 20024               
evict_skip                = 1214                
evict_l2_eligible         = 71168               
p                         = 33800736768         
l2_compress_successes     = 0                   
recycle_miss              = 0                   
l2_writes_done            = 0                   
----------

Thu Dec  4 16:15:24 IST 2014

l2_size                   = 0                   
other_size                = 17752040            
c_max                     = 67601473536         
hash_elements             = 41863               
duplicate_reads           = 0                   
prefetch_data_hits        = 537                 
l2_read_bytes             = 0                   
demand_metadata_misses    = 12492               
prefetch_data_misses      = 1540                
hash_chain_max            = 2                   
l2_hits                   = 0                   
l2_compress_failures      = 0                   
mfu_ghost_hits            = 0                   
l2_rw_clash               = 0                   
memory_throttle_count     = 0                   
crtime                    = 4.537754192         
arc_meta_used             = 99137880            
hash_elements_max         = 43929               
size                      = 6192858456          
hits                      = 968809              
mfu_hits                  = 659725              
prefetch_metadata_hits    = 10297               
hash_chains               = 87                  
evict_l2_cached           = 0                   
l2_writes_sent            = 0                   
l2_writes_error           = 0                   
deleted                   = 15                  
hdr_size                  = 12559216            
l2_compress_zeros         = 0                   
l2_abort_lowmem           = 0                   
l2_io_error               = 0                   
l2_hdr_size               = 0                   
l2_cksum_bad              = 0                   
l2_evict_reading          = 0                   
arc_meta_limit            = 16900368384         
prefetch_metadata_misses  = 3381                
l2_evict_lock_retry       = 0                   
hash_collisions           = 880                 
l2_writes_hdr_miss        = 0                   
evict_l2_ineligible       = 4096                
l2_write_bytes            = 0                   
data_size                 = 6162547200          
c                         = 67601473536         
l2_asize                  = 0                   
class                     = misc                
mutex_miss                = 0                   
snaptime                  = 1097.957144074      
duplicate_buffers_size    = 0                   
l2_misses                 = 0                   
mru_ghost_hits            = 0                   
demand_metadata_hits      = 29094               
arc_meta_max              = 99660136            
c_min                     = 2146100480          
mru_hits                  = 298250              
demand_data_misses        = 2615                
duplicate_buffers         = 0                   
demand_data_hits          = 928881              
l2_free_on_write          = 0                   
l2_feeds                  = 0                   
misses                    = 20028               
evict_skip                = 1214                
evict_l2_eligible         = 71168               
p                         = 33800736768         
l2_compress_successes     = 0                   
recycle_miss              = 0                   
l2_writes_done            = 0                   
----------

Thu Dec  4 16:15:27 IST 2014

l2_size                   = 0                   
other_size                = 17651840            
c_max                     = 67601473536         
hash_elements             = 44052               
duplicate_reads           = 0                   
prefetch_data_hits        = 537                 
l2_read_bytes             = 0                   
demand_metadata_misses    = 12496               
prefetch_data_misses      = 1540                
hash_chain_max            = 2                   
l2_hits                   = 0                   
l2_compress_failures      = 0                   
mfu_ghost_hits            = 0                   
l2_rw_clash               = 0                   
memory_throttle_count     = 0                   
crtime                    = 4.537754192         
arc_meta_used             = 99623408            
hash_elements_max         = 44527               
size                      = 5948239344          
hits                      = 972940              
mfu_hits                  = 661821              
prefetch_metadata_hits    = 10297               
hash_chains               = 92                  
evict_l2_cached           = 0                   
l2_writes_sent            = 0                   
l2_writes_error           = 0                   
deleted                   = 15                  
hdr_size                  = 12559216            
l2_compress_zeros         = 0                   
l2_abort_lowmem           = 0                   
l2_io_error               = 0                   
l2_hdr_size               = 0                   
l2_cksum_bad              = 0                   
l2_evict_reading          = 0                   
arc_meta_limit            = 16900368384         
prefetch_metadata_misses  = 3381                
l2_evict_lock_retry       = 0                   
hash_collisions           = 891                 
l2_writes_hdr_miss        = 0                   
evict_l2_ineligible       = 4096                
l2_write_bytes            = 0                   
data_size                 = 5918028288          
c                         = 67601473536         
l2_asize                  = 0                   
class                     = misc                
mutex_miss                = 0                   
snaptime                  = 1100.964341965      
duplicate_buffers_size    = 0                   
l2_misses                 = 0                   
mru_ghost_hits            = 0                   
demand_metadata_hits      = 29100               
arc_meta_max              = 99660136            
c_min                     = 2146100480          
mru_hits                  = 300285              
demand_data_misses        = 2615                
duplicate_buffers         = 0                   
demand_data_hits          = 933006              
l2_free_on_write          = 0                   
l2_feeds                  = 0                   
misses                    = 20032               
evict_skip                = 1214                
evict_l2_eligible         = 71168               
p                         = 33800736768         
l2_compress_successes     = 0                   
recycle_miss              = 0                   
l2_writes_done            = 0                   
----------

Thu Dec  4 16:15:30 IST 2014

l2_size                   = 0                   
other_size                = 17736840            
c_max                     = 67601473536         
hash_elements             = 39286               
duplicate_reads           = 0                   
prefetch_data_hits        = 537                 
l2_read_bytes             = 0                   
demand_metadata_misses    = 12500               
prefetch_data_misses      = 1540                
hash_chain_max            = 2                   
l2_hits                   = 0                   
l2_compress_failures      = 0                   
mfu_ghost_hits            = 0                   
l2_rw_clash               = 0                   
memory_throttle_count     = 0                   
crtime                    = 4.537754192         
arc_meta_used             = 99134968            
hash_elements_max         = 44527               
size                      = 5866748408          
hits                      = 999296              
mfu_hits                  = 681359              
prefetch_metadata_hits    = 10297               
hash_chains               = 89                  
evict_l2_cached           = 0                   
l2_writes_sent            = 0                   
l2_writes_error           = 0                   
deleted                   = 15                  
hdr_size                  = 12559216            
l2_compress_zeros         = 0                   
l2_abort_lowmem           = 0                   
l2_io_error               = 0                   
l2_hdr_size               = 0                   
l2_cksum_bad              = 0                   
l2_evict_reading          = 0                   
arc_meta_limit            = 16900368384         
prefetch_metadata_misses  = 3381                
l2_evict_lock_retry       = 0                   
hash_collisions           = 922                 
l2_writes_hdr_miss        = 0                   
evict_l2_ineligible       = 4096                
l2_write_bytes            = 0                   
data_size                 = 5836452352          
c                         = 67601473536         
l2_asize                  = 0                   
class                     = misc                
mutex_miss                = 0                   
snaptime                  = 1103.989100171      
duplicate_buffers_size    = 0                   
l2_misses                 = 0                   
mru_ghost_hits            = 0                   
demand_metadata_hits      = 29108               
arc_meta_max              = 99718496            
c_min                     = 2146100480          
mru_hits                  = 307103              
demand_data_misses        = 2615                
duplicate_buffers         = 0                   
demand_data_hits          = 959354              
l2_free_on_write          = 0                   
l2_feeds                  = 0                   
misses                    = 20036               
evict_skip                = 1214                
evict_l2_eligible         = 71168               
p                         = 33800736768         
l2_compress_successes     = 0                   
recycle_miss              = 0                   
l2_writes_done            = 0                   
----------

Thu Dec  4 16:15:33 IST 2014

l2_size                   = 0                   
other_size                = 17708240            
c_max                     = 67601473536         
hash_elements             = 43427               
duplicate_reads           = 0                   
prefetch_data_hits        = 537                 
l2_read_bytes             = 0                   
demand_metadata_misses    = 12500               
prefetch_data_misses      = 1540                
hash_chain_max            = 2                   
l2_hits                   = 0                   
l2_compress_failures      = 0                   
mfu_ghost_hits            = 0                   
l2_rw_clash               = 0                   
memory_throttle_count     = 0                   
crtime                    = 4.537754192         
arc_meta_used             = 99106368            
hash_elements_max         = 44527               
size                      = 5953882688          
hits                      = 1009876             
mfu_hits                  = 687985              
prefetch_metadata_hits    = 10297               
hash_chains               = 107                 
evict_l2_cached           = 0                   
l2_writes_sent            = 0                   
l2_writes_error           = 0                   
deleted                   = 15                  
hdr_size                  = 12559216            
l2_compress_zeros         = 0                   
l2_abort_lowmem           = 0                   
l2_io_error               = 0                   
l2_hdr_size               = 0                   
l2_cksum_bad              = 0                   
l2_evict_reading          = 0                   
arc_meta_limit            = 16900368384         
prefetch_metadata_misses  = 3381                
l2_evict_lock_retry       = 0                   
hash_collisions           = 959                 
l2_writes_hdr_miss        = 0                   
evict_l2_ineligible       = 4096                
l2_write_bytes            = 0                   
data_size                 = 5923615232          
c                         = 67601473536         
l2_asize                  = 0                   
class                     = misc                
mutex_miss                = 0                   
snaptime                  = 1106.99462468       
duplicate_buffers_size    = 0                   
l2_misses                 = 0                   
mru_ghost_hits            = 0                   
demand_metadata_hits      = 29109               
arc_meta_max              = 99718496            
c_min                     = 2146100480          
mru_hits                  = 311057              
demand_data_misses        = 2615                
duplicate_buffers         = 0                   
demand_data_hits          = 969933              
l2_free_on_write          = 0                   
l2_feeds                  = 0                   
misses                    = 20036               
evict_skip                = 1214                
evict_l2_eligible         = 71168               
p                         = 33800736768         
l2_compress_successes     = 0                   
recycle_miss              = 0                   
l2_writes_done            = 0                   
----------

Thu Dec  4 16:15:36 IST 2014

l2_size                   = 0                   
other_size                = 17820440            
c_max                     = 67601473536         
hash_elements             = 38756               
duplicate_reads           = 0                   
prefetch_data_hits        = 537                 
l2_read_bytes             = 0                   
demand_metadata_misses    = 12508               
prefetch_data_misses      = 1540                
hash_chain_max            = 2                   
l2_hits                   = 0                   
l2_compress_failures      = 0                   
mfu_ghost_hits            = 0                   
l2_rw_clash               = 0                   
memory_throttle_count     = 0                   
crtime                    = 4.537754192         
arc_meta_used             = 99222664            
hash_elements_max         = 44527               
size                      = 5931061384          
hits                      = 1034158             
mfu_hits                  = 705727              
prefetch_metadata_hits    = 10297               
hash_chains               = 79                  
evict_l2_cached           = 0                   
l2_writes_sent            = 0                   
l2_writes_error           = 0                   
deleted                   = 15                  
hdr_size                  = 12559216            
l2_compress_zeros         = 0                   
l2_abort_lowmem           = 0                   
l2_io_error               = 0                   
l2_hdr_size               = 0                   
l2_cksum_bad              = 0                   
l2_evict_reading          = 0                   
arc_meta_limit            = 16900368384         
prefetch_metadata_misses  = 3381                
l2_evict_lock_retry       = 0                   
hash_collisions           = 970                 
l2_writes_hdr_miss        = 0                   
evict_l2_ineligible       = 4096                
l2_write_bytes            = 0                   
data_size                 = 5900681728          
c                         = 67601473536         
l2_asize                  = 0                   
class                     = misc                
mutex_miss                = 0                   
snaptime                  = 1109.998397046      
duplicate_buffers_size    = 0                   
l2_misses                 = 0                   
mru_ghost_hits            = 0                   
demand_metadata_hits      = 29126               
arc_meta_max              = 99816784            
c_min                     = 2146100480          
mru_hits                  = 317597              
demand_data_misses        = 2615                
duplicate_buffers         = 0                   
demand_data_hits          = 994198              
l2_free_on_write          = 0                   
l2_feeds                  = 0                   
misses                    = 20044               
evict_skip                = 1214                
evict_l2_eligible         = 71168               
p                         = 33800736768         
l2_compress_successes     = 0                   
recycle_miss              = 0                   
l2_writes_done            = 0                   
----------

Thu Dec  4 16:15:39 IST 2014

l2_size                   = 0                   
other_size                = 17808640            
c_max                     = 67601473536         
hash_elements             = 44706               
duplicate_reads           = 0                   
prefetch_data_hits        = 537                 
l2_read_bytes             = 0                   
demand_metadata_misses    = 12508               
prefetch_data_misses      = 1540                
hash_chain_max            = 2                   
l2_hits                   = 0                   
l2_compress_failures      = 0                   
mfu_ghost_hits            = 0                   
l2_rw_clash               = 0                   
memory_throttle_count     = 0                   
crtime                    = 4.537754192         
arc_meta_used             = 99212592            
hash_elements_max         = 44706               
size                      = 6196209968          
hits                      = 1039940             
mfu_hits                  = 709638              
prefetch_metadata_hits    = 10297               
hash_chains               = 103                 
evict_l2_cached           = 0                   
l2_writes_sent            = 0                   
l2_writes_error           = 0                   
deleted                   = 15                  
hdr_size                  = 12560944            
l2_compress_zeros         = 0                   
l2_abort_lowmem           = 0                   
l2_io_error               = 0                   
l2_hdr_size               = 0                   
l2_cksum_bad              = 0                   
l2_evict_reading          = 0                   
arc_meta_limit            = 16900368384         
prefetch_metadata_misses  = 3381                
l2_evict_lock_retry       = 0                   
hash_collisions           = 1014                
l2_writes_hdr_miss        = 0                   
evict_l2_ineligible       = 4096                
l2_write_bytes            = 0                   
data_size                 = 6165840384          
c                         = 67601473536         
l2_asize                  = 0                   
class                     = misc                
mutex_miss                = 0                   
snaptime                  = 1113.002157194      
duplicate_buffers_size    = 0                   
l2_misses                 = 0                   
mru_ghost_hits            = 0                   
demand_metadata_hits      = 29126               
arc_meta_max              = 99816784            
c_min                     = 2146100480          
mru_hits                  = 319468              
demand_data_misses        = 2615                
duplicate_buffers         = 0                   
demand_data_hits          = 999980              
l2_free_on_write          = 0                   
l2_feeds                  = 0                   
misses                    = 20044               
evict_skip                = 1214                
evict_l2_eligible         = 71168               
p                         = 33800736768         
l2_compress_successes     = 0                   
recycle_miss              = 0                   
l2_writes_done            = 0                   
----------

Thu Dec  4 16:15:42 IST 2014

l2_size                   = 0                   
other_size                = 17820240            
c_max                     = 67601473536         
hash_elements             = 42808               
duplicate_reads           = 0                   
prefetch_data_hits        = 537                 
l2_read_bytes             = 0                   
demand_metadata_misses    = 12516               
prefetch_data_misses      = 1540                
hash_chain_max            = 2                   
l2_hits                   = 0                   
l2_compress_failures      = 0                   
mfu_ghost_hits            = 0                   
l2_rw_clash               = 0                   
memory_throttle_count     = 0                   
crtime                    = 4.537754192         
arc_meta_used             = 99240576            
hash_elements_max         = 46675               
size                      = 6415390336          
hits                      = 1059255             
mfu_hits                  = 723283              
prefetch_metadata_hits    = 10297               
hash_chains               = 108                 
evict_l2_cached           = 0                   
l2_writes_sent            = 0                   
l2_writes_error           = 0                   
deleted                   = 15                  
hdr_size                  = 12560944            
l2_compress_zeros         = 0                   
l2_abort_lowmem           = 0                   
l2_io_error               = 0                   
l2_hdr_size               = 0                   
l2_cksum_bad              = 0                   
l2_evict_reading          = 0                   
arc_meta_limit            = 16900368384         
prefetch_metadata_misses  = 3381                
l2_evict_lock_retry       = 0                   
hash_collisions           = 1042                
l2_writes_hdr_miss        = 0                   
evict_l2_ineligible       = 4096                
l2_write_bytes            = 0                   
data_size                 = 6385009152          
c                         = 67601473536         
l2_asize                  = 0                   
class                     = misc                
mutex_miss                = 0                   
snaptime                  = 1116.00772062       
duplicate_buffers_size    = 0                   
l2_misses                 = 0                   
mru_ghost_hits            = 0                   
demand_metadata_hits      = 29140               
arc_meta_max              = 99824704            
c_min                     = 2146100480          
mru_hits                  = 325138              
demand_data_misses        = 2615                
duplicate_buffers         = 0                   
demand_data_hits          = 1019281             
l2_free_on_write          = 0                   
l2_feeds                  = 0                   
misses                    = 20052               
evict_skip                = 1214                
evict_l2_eligible         = 71168               
p                         = 33800736768         
l2_compress_successes     = 0                   
recycle_miss              = 0                   
l2_writes_done            = 0                   
----------

Thu Dec  4 16:15:45 IST 2014

l2_size                   = 0                   
other_size                = 17674040            
c_max                     = 67601473536         
hash_elements             = 44891               
duplicate_reads           = 0                   
prefetch_data_hits        = 537                 
l2_read_bytes             = 0                   
demand_metadata_misses    = 12516               
prefetch_data_misses      = 1540                
hash_chain_max            = 2                   
l2_hits                   = 0                   
l2_compress_failures      = 0                   
mfu_ghost_hits            = 0                   
l2_rw_clash               = 0                   
memory_throttle_count     = 0                   
crtime                    = 4.537754192         
arc_meta_used             = 99094376            
hash_elements_max         = 46675               
size                      = 5900262248          
hits                      = 1068199             
mfu_hits                  = 728474              
prefetch_metadata_hits    = 10297               
hash_chains               = 130                 
evict_l2_cached           = 0                   
l2_writes_sent            = 0                   
l2_writes_error           = 0                   
deleted                   = 15                  
hdr_size                  = 12560944            
l2_compress_zeros         = 0                   
l2_abort_lowmem           = 0                   
l2_io_error               = 0                   
l2_hdr_size               = 0                   
l2_cksum_bad              = 0                   
l2_evict_reading          = 0                   
arc_meta_limit            = 16900368384         
prefetch_metadata_misses  = 3381                
l2_evict_lock_retry       = 0                   
hash_collisions           = 1081                
l2_writes_hdr_miss        = 0                   
evict_l2_ineligible       = 4096                
l2_write_bytes            = 0                   
data_size                 = 5870027264          
c                         = 67601473536         
l2_asize                  = 0                   
class                     = misc                
mutex_miss                = 0                   
snaptime                  = 1119.021235208      
duplicate_buffers_size    = 0                   
l2_misses                 = 0                   
mru_ghost_hits            = 0                   
demand_metadata_hits      = 29140               
arc_meta_max              = 99824704            
c_min                     = 2146100480          
mru_hits                  = 328891              
demand_data_misses        = 2615                
duplicate_buffers         = 0                   
demand_data_hits          = 1028225             
l2_free_on_write          = 0                   
l2_feeds                  = 0                   
misses                    = 20052               
evict_skip                = 1214                
evict_l2_eligible         = 71168               
p                         = 33800736768         
l2_compress_successes     = 0                   
recycle_miss              = 0                   
l2_writes_done            = 0                   
----------

Thu Dec  4 16:15:48 IST 2014

l2_size                   = 0                   
other_size                = 17766200            
c_max                     = 67601473536         
hash_elements             = 39994               
duplicate_reads           = 0                   
prefetch_data_hits        = 537                 
l2_read_bytes             = 0                   
demand_metadata_misses    = 12524               
prefetch_data_misses      = 1540                
hash_chain_max            = 2                   
l2_hits                   = 0                   
l2_compress_failures      = 0                   
mfu_ghost_hits            = 0                   
l2_rw_clash               = 0                   
memory_throttle_count     = 0                   
crtime                    = 4.537754192         
arc_meta_used             = 99751784            
hash_elements_max         = 46675               
size                      = 5771420520          
hits                      = 1088758             
mfu_hits                  = 743653              
prefetch_metadata_hits    = 10297               
hash_chains               = 94                  
evict_l2_cached           = 0                   
l2_writes_sent            = 0                   
l2_writes_error           = 0                   
deleted                   = 15                  
hdr_size                  = 12560944            
l2_compress_zeros         = 0                   
l2_abort_lowmem           = 0                   
l2_io_error               = 0                   
l2_hdr_size               = 0                   
l2_cksum_bad              = 0                   
l2_evict_reading          = 0                   
arc_meta_limit            = 16900368384         
prefetch_metadata_misses  = 3381                
l2_evict_lock_retry       = 0                   
hash_collisions           = 1092                
l2_writes_hdr_miss        = 0                   
evict_l2_ineligible       = 4096                
l2_write_bytes            = 0                   
data_size                 = 5741093376          
c                         = 67601473536         
l2_asize                  = 0                   
class                     = misc                
mutex_miss                = 0                   
snaptime                  = 1122.026835395      
duplicate_buffers_size    = 0                   
l2_misses                 = 0                   
mru_ghost_hits            = 0                   
demand_metadata_hits      = 29147               
arc_meta_max              = 99824704            
c_min                     = 2146100480          
mru_hits                  = 334271              
demand_data_misses        = 2615                
duplicate_buffers         = 0                   
demand_data_hits          = 1048777             
l2_free_on_write          = 0                   
l2_feeds                  = 0                   
misses                    = 20060               
evict_skip                = 1214                
evict_l2_eligible         = 71168               
p                         = 33800736768         
l2_compress_successes     = 0                   
recycle_miss              = 0                   
l2_writes_done            = 0                   
----------

Thu Dec  4 16:15:51 IST 2014

l2_size                   = 0                   
other_size                = 17829240            
c_max                     = 67601473536         
hash_elements             = 43380               
duplicate_reads           = 0                   
prefetch_data_hits        = 537                 
l2_read_bytes             = 0                   
demand_metadata_misses    = 12524               
prefetch_data_misses      = 1540                
hash_chain_max            = 2                   
l2_hits                   = 0                   
l2_compress_failures      = 0                   
mfu_ghost_hits            = 0                   
l2_rw_clash               = 0                   
memory_throttle_count     = 0                   
crtime                    = 4.537754192         
arc_meta_used             = 99249576            
hash_elements_max         = 46675               
size                      = 6035421608          
hits                      = 1096297             
mfu_hits                  = 749091              
prefetch_metadata_hits    = 10297               
hash_chains               = 87                  
evict_l2_cached           = 0                   
l2_writes_sent            = 0                   
l2_writes_error           = 0                   
deleted                   = 15                  
hdr_size                  = 12560944            
l2_compress_zeros         = 0                   
l2_abort_lowmem           = 0                   
l2_io_error               = 0                   
l2_hdr_size               = 0                   
l2_cksum_bad              = 0                   
l2_evict_reading          = 0                   
arc_meta_limit            = 16900368384         
prefetch_metadata_misses  = 3381                
l2_evict_lock_retry       = 0                   
hash_collisions           = 1102                
l2_writes_hdr_miss        = 0                   
evict_l2_ineligible       = 4096                
l2_write_bytes            = 0                   
data_size                 = 6005031424          
c                         = 67601473536         
l2_asize                  = 0                   
class                     = misc                
mutex_miss                = 0                   
snaptime                  = 1125.031042354      
duplicate_buffers_size    = 0                   
l2_misses                 = 0                   
mru_ghost_hits            = 0                   
demand_metadata_hits      = 29154               
arc_meta_max              = 99829992            
c_min                     = 2146100480          
mru_hits                  = 336372              
demand_data_misses        = 2615                
duplicate_buffers         = 0                   
demand_data_hits          = 1056309             
l2_free_on_write          = 0                   
l2_feeds                  = 0                   
misses                    = 20060               
evict_skip                = 1214                
evict_l2_eligible         = 71168               
p                         = 33800736768         
l2_compress_successes     = 0                   
recycle_miss              = 0                   
l2_writes_done            = 0                   
----------

Thu Dec  4 16:15:54 IST 2014

l2_size                   = 0                   
other_size                = 17900600            
c_max                     = 67601473536         
hash_elements             = 39997               
duplicate_reads           = 0                   
prefetch_data_hits        = 537                 
l2_read_bytes             = 0                   
demand_metadata_misses    = 12528               
prefetch_data_misses      = 1540                
hash_chain_max            = 2                   
l2_hits                   = 0                   
l2_compress_failures      = 0                   
mfu_ghost_hits            = 0                   
l2_rw_clash               = 0                   
memory_throttle_count     = 0                   
crtime                    = 4.537754192         
arc_meta_used             = 99390568            
hash_elements_max         = 46675               
size                      = 5948399720          
hits                      = 1120591             
mfu_hits                  = 765902              
prefetch_metadata_hits    = 10297               
hash_chains               = 83                  
evict_l2_cached           = 0                   
l2_writes_sent            = 0                   
l2_writes_error           = 0                   
deleted                   = 15                  
hdr_size                  = 12560944            
l2_compress_zeros         = 0                   
l2_abort_lowmem           = 0                   
l2_io_error               = 0                   
l2_hdr_size               = 0                   
l2_cksum_bad              = 0                   
l2_evict_reading          = 0                   
arc_meta_limit            = 16900368384         
prefetch_metadata_misses  = 3381                
l2_evict_lock_retry       = 0                   
hash_collisions           = 1118                
l2_writes_hdr_miss        = 0                   
evict_l2_ineligible       = 4096                
l2_write_bytes            = 0                   
data_size                 = 5917938176          
c                         = 67601473536         
l2_asize                  = 0                   
class                     = misc                
mutex_miss                = 0                   
snaptime                  = 1128.034852134      
duplicate_buffers_size    = 0                   
l2_misses                 = 0                   
mru_ghost_hits            = 0                   
demand_metadata_hits      = 29163               
arc_meta_max              = 99971728            
c_min                     = 2146100480          
mru_hits                  = 343855              
demand_data_misses        = 2615                
duplicate_buffers         = 0                   
demand_data_hits          = 1080594             
l2_free_on_write          = 0                   
l2_feeds                  = 0                   
misses                    = 20064               
evict_skip                = 1214                
evict_l2_eligible         = 71168               
p                         = 33800736768         
l2_compress_successes     = 0                   
recycle_miss              = 0                   
l2_writes_done            = 0                   
----------

Thu Dec  4 16:15:57 IST 2014

l2_size                   = 0                   
other_size                = 17931600            
c_max                     = 67601473536         
hash_elements             = 44425               
duplicate_reads           = 0                   
prefetch_data_hits        = 537                 
l2_read_bytes             = 0                   
demand_metadata_misses    = 12532               
prefetch_data_misses      = 1540                
hash_chain_max            = 2                   
l2_hits                   = 0                   
l2_compress_failures      = 0                   
mfu_ghost_hits            = 0                   
l2_rw_clash               = 0                   
memory_throttle_count     = 0                   
crtime                    = 4.537754192         
arc_meta_used             = 100027776           
hash_elements_max         = 46675               
size                      = 5889137024          
hits                      = 1123794             
mfu_hits                  = 768068              
prefetch_metadata_hits    = 10297               
hash_chains               = 102                 
evict_l2_cached           = 0                   
l2_writes_sent            = 0                   
l2_writes_error           = 0                   
deleted                   = 15                  
hdr_size                  = 12560944            
l2_compress_zeros         = 0                   
l2_abort_lowmem           = 0                   
l2_io_error               = 0                   
l2_hdr_size               = 0                   
l2_cksum_bad              = 0                   
l2_evict_reading          = 0                   
arc_meta_limit            = 16900368384         
prefetch_metadata_misses  = 3381                
l2_evict_lock_retry       = 0                   
hash_collisions           = 1137                
l2_writes_hdr_miss        = 0                   
evict_l2_ineligible       = 4096                
l2_write_bytes            = 0                   
data_size                 = 5858644480          
c                         = 67601473536         
l2_asize                  = 0                   
class                     = misc                
mutex_miss                = 0                   
snaptime                  = 1131.040711699      
duplicate_buffers_size    = 0                   
l2_misses                 = 0                   
mru_ghost_hits            = 0                   
demand_metadata_hits      = 29170               
arc_meta_max              = 99994208            
c_min                     = 2146100480          
mru_hits                  = 344892              
demand_data_misses        = 2615                
duplicate_buffers         = 0                   
demand_data_hits          = 1083790             
l2_free_on_write          = 0                   
l2_feeds                  = 0                   
misses                    = 20068               
evict_skip                = 1214                
evict_l2_eligible         = 71168               
p                         = 33800736768         
l2_compress_successes     = 0                   
recycle_miss              = 0                   
l2_writes_done            = 0                   
----------

Thu Dec  4 16:16:00 IST 2014

l2_size                   = 0                   
other_size                = 17803560            
c_max                     = 67601473536         
hash_elements             = 42523               
duplicate_reads           = 0                   
prefetch_data_hits        = 537                 
l2_read_bytes             = 0                   
demand_metadata_misses    = 12532               
prefetch_data_misses      = 1540                
hash_chain_max            = 2                   
l2_hits                   = 0                   
l2_compress_failures      = 0                   
mfu_ghost_hits            = 0                   
l2_rw_clash               = 0                   
memory_throttle_count     = 0                   
crtime                    = 4.537754192         
arc_meta_used             = 99334488            
hash_elements_max         = 46675               
size                      = 5995791704          
hits                      = 1137123             
mfu_hits                  = 776941              
prefetch_metadata_hits    = 10297               
hash_chains               = 84                  
evict_l2_cached           = 0                   
l2_writes_sent            = 0                   
l2_writes_error           = 0                   
deleted                   = 15                  
hdr_size                  = 12560944            
l2_compress_zeros         = 0                   
l2_abort_lowmem           = 0                   
l2_io_error               = 0                   
l2_hdr_size               = 0                   
l2_cksum_bad              = 0                   
l2_evict_reading          = 0                   
arc_meta_limit            = 16900368384         
prefetch_metadata_misses  = 3381                
l2_evict_lock_retry       = 0                   
hash_collisions           = 1145                
l2_writes_hdr_miss        = 0                   
evict_l2_ineligible       = 4096                
l2_write_bytes            = 0                   
data_size                 = 5965427200          
c                         = 67601473536         
l2_asize                  = 0                   
class                     = misc                
mutex_miss                = 0                   
snaptime                  = 1134.044734783      
duplicate_buffers_size    = 0                   
l2_misses                 = 0                   
mru_ghost_hits            = 0                   
demand_metadata_hits      = 29171               
arc_meta_max              = 100042776           
c_min                     = 2146100480          
mru_hits                  = 349348              
demand_data_misses        = 2615                
duplicate_buffers         = 0                   
demand_data_hits          = 1097118             
l2_free_on_write          = 0                   
l2_feeds                  = 0                   
misses                    = 20068               
evict_skip                = 1214                
evict_l2_eligible         = 71168               
p                         = 33800736768         
l2_compress_successes     = 0                   
recycle_miss              = 0                   
l2_writes_done            = 0                   
----------

Thu Dec  4 16:16:03 IST 2014

l2_size                   = 0                   
other_size                = 18005000            
c_max                     = 67601473536         
hash_elements             = 40668               
duplicate_reads           = 0                   
prefetch_data_hits        = 537                 
l2_read_bytes             = 0                   
demand_metadata_misses    = 12540               
prefetch_data_misses      = 1540                
hash_chain_max            = 2                   
l2_hits                   = 0                   
l2_compress_failures      = 0                   
mfu_ghost_hits            = 0                   
l2_rw_clash               = 0                   
memory_throttle_count     = 0                   
crtime                    = 4.537754192         
arc_meta_used             = 100146232           
hash_elements_max         = 46675               
size                      = 5979957304          
hits                      = 1157458             
mfu_hits                  = 791193              
prefetch_metadata_hits    = 10297               
hash_chains               = 85                  
evict_l2_cached           = 0                   
l2_writes_sent            = 0                   
l2_writes_error           = 0                   
deleted                   = 15                  
hdr_size                  = 12560944            
l2_compress_zeros         = 0                   
l2_abort_lowmem           = 0                   
l2_io_error               = 0                   
l2_hdr_size               = 0                   
l2_cksum_bad              = 0                   
l2_evict_reading          = 0                   
arc_meta_limit            = 16900368384         
prefetch_metadata_misses  = 3381                
l2_evict_lock_retry       = 0                   
hash_collisions           = 1162                
l2_writes_hdr_miss        = 0                   
evict_l2_ineligible       = 4096                
l2_write_bytes            = 0                   
data_size                 = 5949391360          
c                         = 67601473536         
l2_asize                  = 0                   
class                     = misc                
mutex_miss                = 0                   
snaptime                  = 1137.04862393       
duplicate_buffers_size    = 0                   
l2_misses                 = 0                   
mru_ghost_hits            = 0                   
demand_metadata_hits      = 29187               
arc_meta_max              = 100145632           
c_min                     = 2146100480          
mru_hits                  = 355431              
demand_data_misses        = 2615                
duplicate_buffers         = 0                   
demand_data_hits          = 1117437             
l2_free_on_write          = 0                   
l2_feeds                  = 0                   
misses                    = 20076               
evict_skip                = 1214                
evict_l2_eligible         = 71168               
p                         = 33800736768         
l2_compress_successes     = 0                   
recycle_miss              = 0                   
l2_writes_done            = 0                   
----------

Thu Dec  4 16:16:06 IST 2014

l2_size                   = 0                   
other_size                = 18052040            
c_max                     = 67601473536         
hash_elements             = 45931               
duplicate_reads           = 0                   
prefetch_data_hits        = 537                 
l2_read_bytes             = 0                   
demand_metadata_misses    = 12540               
prefetch_data_misses      = 1540                
hash_chain_max            = 2                   
l2_hits                   = 0                   
l2_compress_failures      = 0                   
mfu_ghost_hits            = 0                   
l2_rw_clash               = 0                   
memory_throttle_count     = 0                   
crtime                    = 4.537754192         
arc_meta_used             = 99615736            
hash_elements_max         = 46725               
size                      = 6264246264          
hits                      = 1168001             
mfu_hits                  = 798095              
prefetch_metadata_hits    = 10297               
hash_chains               = 98                  
evict_l2_cached           = 0                   
l2_writes_sent            = 0                   
l2_writes_error           = 0                   
deleted                   = 15                  
hdr_size                  = 12560944            
l2_compress_zeros         = 0                   
l2_abort_lowmem           = 0                   
l2_io_error               = 0                   
l2_hdr_size               = 0                   
l2_cksum_bad              = 0                   
l2_evict_reading          = 0                   
arc_meta_limit            = 16900368384         
prefetch_metadata_misses  = 3381                
l2_evict_lock_retry       = 0                   
hash_collisions           = 1192                
l2_writes_hdr_miss        = 0                   
evict_l2_ineligible       = 4096                
l2_write_bytes            = 0                   
data_size                 = 6233633280          
c                         = 67601473536         
l2_asize                  = 0                   
class                     = misc                
mutex_miss                = 0                   
snaptime                  = 1140.052771613      
duplicate_buffers_size    = 0                   
l2_misses                 = 0                   
mru_ghost_hits            = 0                   
demand_metadata_hits      = 29188               
arc_meta_max              = 100153872           
c_min                     = 2146100480          
mru_hits                  = 359072              
demand_data_misses        = 2615                
duplicate_buffers         = 0                   
demand_data_hits          = 1127979             
l2_free_on_write          = 0                   
l2_feeds                  = 0                   
misses                    = 20076               
evict_skip                = 1214                
evict_l2_eligible         = 71168               
p                         = 33800736768         
l2_compress_successes     = 0                   
recycle_miss              = 0                   
l2_writes_done            = 0                   
----------

Thu Dec  4 16:16:09 IST 2014

l2_size                   = 0                   
other_size                = 18144240            
c_max                     = 67601473536         
hash_elements             = 44399               
duplicate_reads           = 0                   
prefetch_data_hits        = 537                 
l2_read_bytes             = 0                   
demand_metadata_misses    = 12548               
prefetch_data_misses      = 1540                
hash_chain_max            = 2                   
l2_hits                   = 0                   
l2_compress_failures      = 0                   
mfu_ghost_hits            = 0                   
l2_rw_clash               = 0                   
memory_throttle_count     = 0                   
crtime                    = 4.537754192         
arc_meta_used             = 99752992            
hash_elements_max         = 47118               
size                      = 6250227744          
hits                      = 1186886             
mfu_hits                  = 811377              
prefetch_metadata_hits    = 10297               
hash_chains               = 155                 
evict_l2_cached           = 0                   
l2_writes_sent            = 0                   
l2_writes_error           = 0                   
deleted                   = 15                  
hdr_size                  = 12560944            
l2_compress_zeros         = 0                   
l2_abort_lowmem           = 0                   
l2_io_error               = 0                   
l2_hdr_size               = 0                   
l2_cksum_bad              = 0                   
l2_evict_reading          = 0                   
arc_meta_limit            = 16900368384         
prefetch_metadata_misses  = 3381                
l2_evict_lock_retry       = 0                   
hash_collisions           = 1302                
l2_writes_hdr_miss        = 0                   
evict_l2_ineligible       = 4096                
l2_write_bytes            = 0                   
data_size                 = 6219522560          
c                         = 67601473536         
l2_asize                  = 0                   
class                     = misc                
mutex_miss                = 0                   
snaptime                  = 1143.058190189      
duplicate_buffers_size    = 0                   
l2_misses                 = 0                   
mru_ghost_hits            = 0                   
demand_metadata_hits      = 29202               
arc_meta_max              = 100285688           
c_min                     = 2146100480          
mru_hits                  = 364675              
demand_data_misses        = 2615                
duplicate_buffers         = 0                   
demand_data_hits          = 1146850             
l2_free_on_write          = 0                   
l2_feeds                  = 0                   
misses                    = 20084               
evict_skip                = 1214                
evict_l2_eligible         = 71168               
p                         = 33800736768         
l2_compress_successes     = 0                   
recycle_miss              = 0                   
l2_writes_done            = 0                   
----------

Thu Dec  4 16:16:12 IST 2014

l2_size                   = 0                   
other_size                = 18053440            
c_max                     = 67601473536         
hash_elements             = 46005               
duplicate_reads           = 0                   
prefetch_data_hits        = 537                 
l2_read_bytes             = 0                   
demand_metadata_misses    = 12552               
prefetch_data_misses      = 1540                
hash_chain_max            = 2                   
l2_hits                   = 0                   
l2_compress_failures      = 0                   
mfu_ghost_hits            = 0                   
l2_rw_clash               = 0                   
memory_throttle_count     = 0                   
crtime                    = 4.537754192         
arc_meta_used             = 100254576           
hash_elements_max         = 47118               
size                      = 6189538160          
hits                      = 1194015             
mfu_hits                  = 815523              
prefetch_metadata_hits    = 10297               
hash_chains               = 129                 
evict_l2_cached           = 0                   
l2_writes_sent            = 0                   
l2_writes_error           = 0                   
deleted                   = 15                  
hdr_size                  = 12560944            
l2_compress_zeros         = 0                   
l2_abort_lowmem           = 0                   
l2_io_error               = 0                   
l2_hdr_size               = 0                   
l2_cksum_bad              = 0                   
l2_evict_reading          = 0                   
arc_meta_limit            = 16900368384         
prefetch_metadata_misses  = 3381                
l2_evict_lock_retry       = 0                   
hash_collisions           = 1311                
l2_writes_hdr_miss        = 0                   
evict_l2_ineligible       = 397312              
l2_write_bytes            = 0                   
data_size                 = 6158923776          
c                         = 6190066904          
l2_asize                  = 0                   
class                     = misc                
mutex_miss                = 0                   
snaptime                  = 1146.06243032       
duplicate_buffers_size    = 0                   
l2_misses                 = 0                   
mru_ghost_hits            = 0                   
demand_metadata_hits      = 29214               
arc_meta_max              = 100285688           
c_min                     = 2146100480          
mru_hits                  = 367658              
demand_data_misses        = 2615                
duplicate_buffers         = 0                   
demand_data_hits          = 1153967             
l2_free_on_write          = 0                   
l2_feeds                  = 0                   
misses                    = 20088               
evict_skip                = 1214                
evict_l2_eligible         = 1011712             
p                         = 3096114796          
l2_compress_successes     = 0                   
recycle_miss              = 1                   
l2_writes_done            = 0                   
----------

Thu Dec  4 16:16:15 IST 2014

l2_size                   = 0                   
other_size                = 17695320            
c_max                     = 67601473536         
hash_elements             = 44156               
duplicate_reads           = 0                   
prefetch_data_hits        = 537                 
l2_read_bytes             = 0                   
demand_metadata_misses    = 12556               
prefetch_data_misses      = 1540                
hash_chain_max            = 2                   
l2_hits                   = 0                   
l2_compress_failures      = 0                   
mfu_ghost_hits            = 0                   
l2_rw_clash               = 0                   
memory_throttle_count     = 0                   
crtime                    = 4.537754192         
arc_meta_used             = 99370392            
hash_elements_max         = 47845               
size                      = 6062509464          
hits                      = 1214370             
mfu_hits                  = 831061              
prefetch_metadata_hits    = 10297               
hash_chains               = 121                 
evict_l2_cached           = 0                   
l2_writes_sent            = 0                   
l2_writes_error           = 0                   
deleted                   = 15                  
hdr_size                  = 12583744            
l2_compress_zeros         = 0                   
l2_abort_lowmem           = 0                   
l2_io_error               = 0                   
l2_hdr_size               = 0                   
l2_cksum_bad              = 0                   
l2_evict_reading          = 0                   
arc_meta_limit            = 16900368384         
prefetch_metadata_misses  = 3381                
l2_evict_lock_retry       = 0                   
hash_collisions           = 1369                
l2_writes_hdr_miss        = 0                   
evict_l2_ineligible       = 132255744           
l2_write_bytes            = 0                   
data_size                 = 6032230400          
c                         = 6190066904          
l2_asize                  = 0                   
class                     = misc                
mutex_miss                = 1                   
snaptime                  = 1149.080481297      
duplicate_buffers_size    = 0                   
l2_misses                 = 0                   
mru_ghost_hits            = 0                   
demand_metadata_hits      = 29222               
arc_meta_max              = 100285688           
c_min                     = 2146100480          
mru_hits                  = 372475              
demand_data_misses        = 2615                
duplicate_buffers         = 0                   
demand_data_hits          = 1174314             
l2_free_on_write          = 0                   
l2_feeds                  = 0                   
misses                    = 20092               
evict_skip                = 1214                
evict_l2_eligible         = 285229056           
p                         = 3401023084          
l2_compress_successes     = 0                   
recycle_miss              = 219                 
l2_writes_done            = 0                   
----------

Thu Dec  4 16:16:18 IST 2014

l2_size                   = 0                   
other_size                = 17003120            
c_max                     = 67601473536         
hash_elements             = 49196               
duplicate_reads           = 0                   
prefetch_data_hits        = 537                 
l2_read_bytes             = 0                   
demand_metadata_misses    = 12556               
prefetch_data_misses      = 1540                
hash_chain_max            = 2                   
l2_hits                   = 0                   
l2_compress_failures      = 0                   
mfu_ghost_hits            = 0                   
l2_rw_clash               = 0                   
memory_throttle_count     = 0                   
crtime                    = 4.537754192         
arc_meta_used             = 97914576            
hash_elements_max         = 49196               
size                      = 5597583056          
hits                      = 1214720             
mfu_hits                  = 831312              
prefetch_metadata_hits    = 10297               
hash_chains               = 158                 
evict_l2_cached           = 0                   
l2_writes_sent            = 0                   
l2_writes_error           = 0                   
deleted                   = 15                  
hdr_size                  = 11820128            
l2_compress_zeros         = 0                   
l2_abort_lowmem           = 0                   
l2_io_error               = 0                   
l2_hdr_size               = 0                   
l2_cksum_bad              = 0                   
l2_evict_reading          = 0                   
arc_meta_limit            = 16900368384         
prefetch_metadata_misses  = 3381                
l2_evict_lock_retry       = 0                   
hash_collisions           = 1414                
l2_writes_hdr_miss        = 0                   
evict_l2_ineligible       = 132255744           
l2_write_bytes            = 0                   
data_size                 = 5568759808          
c                         = 5859249032          
l2_asize                  = 0                   
class                     = misc                
mutex_miss                = 1                   
snaptime                  = 1152.093743051      
duplicate_buffers_size    = 0                   
l2_misses                 = 0                   
mru_ghost_hits            = 0                   
demand_metadata_hits      = 29222               
arc_meta_max              = 100285688           
c_min                     = 2146100480          
mru_hits                  = 372574              
demand_data_misses        = 2615                
duplicate_buffers         = 0                   
demand_data_hits          = 1174664             
l2_free_on_write          = 0                   
l2_feeds                  = 0                   
misses                    = 20092               
evict_skip                = 1214                
evict_l2_eligible         = 285491200           
p                         = 3295003257          
l2_compress_successes     = 0                   
recycle_miss              = 219                 
l2_writes_done            = 0                   
----------

Thu Dec  4 16:16:21 IST 2014

l2_size                   = 0                   
other_size                = 15860920            
c_max                     = 67601473536         
hash_elements             = 44130               
duplicate_reads           = 0                   
prefetch_data_hits        = 537                 
l2_read_bytes             = 0                   
demand_metadata_misses    = 12564               
prefetch_data_misses      = 1540                
hash_chain_max            = 2                   
l2_hits                   = 0                   
l2_compress_failures      = 0                   
mfu_ghost_hits            = 0                   
l2_rw_clash               = 0                   
memory_throttle_count     = 0                   
crtime                    = 4.537754192         
arc_meta_used             = 91144424            
hash_elements_max         = 49670               
size                      = 4804212968          
hits                      = 1244167             
mfu_hits                  = 851345              
prefetch_metadata_hits    = 10297               
hash_chains               = 102                 
evict_l2_cached           = 0                   
l2_writes_sent            = 0                   
l2_writes_error           = 0                   
deleted                   = 1063                
hdr_size                  = 11332144            
l2_compress_zeros         = 0                   
l2_abort_lowmem           = 0                   
l2_io_error               = 0                   
l2_hdr_size               = 0                   
l2_cksum_bad              = 0                   
l2_evict_reading          = 0                   
arc_meta_limit            = 16900368384         
prefetch_metadata_misses  = 3381                
l2_evict_lock_retry       = 0                   
hash_collisions           = 1433                
l2_writes_hdr_miss        = 0                   
evict_l2_ineligible       = 137154560           
l2_write_bytes            = 0                   
data_size                 = 4777019904          
c                         = 4879960441          
l2_asize                  = 0                   
class                     = misc                
mutex_miss                = 9                   
snaptime                  = 1155.099409293      
duplicate_buffers_size    = 0                   
l2_misses                 = 0                   
mru_ghost_hits            = 24                  
demand_metadata_hits      = 29238               
arc_meta_max              = 100285688           
c_min                     = 2146100480          
mru_hits                  = 381988              
demand_data_misses        = 2639                
duplicate_buffers         = 0                   
demand_data_hits          = 1204095             
l2_free_on_write          = 0                   
l2_feeds                  = 0                   
misses                    = 20124               
evict_skip                = 1214                
evict_l2_eligible         = 1330413056          
p                         = 3274886866          
l2_compress_successes     = 0                   
recycle_miss              = 519                 
l2_writes_done            = 0                   
----------

Thu Dec  4 16:16:24 IST 2014

l2_size                   = 0                   
other_size                = 15703800            
c_max                     = 67601473536         
hash_elements             = 41125               
duplicate_reads           = 0                   
prefetch_data_hits        = 537                 
l2_read_bytes             = 0                   
demand_metadata_misses    = 12572               
prefetch_data_misses      = 1540                
hash_chain_max            = 2                   
l2_hits                   = 0                   
l2_compress_failures      = 0                   
mfu_ghost_hits            = 0                   
l2_rw_clash               = 0                   
memory_throttle_count     = 0                   
crtime                    = 4.537754192         
arc_meta_used             = 85702920            
hash_elements_max         = 49670               
size                      = 4396642568          
hits                      = 1257727             
mfu_hits                  = 860930              
prefetch_metadata_hits    = 10297               
hash_chains               = 92                  
evict_l2_cached           = 0                   
l2_writes_sent            = 0                   
l2_writes_error           = 0                   
deleted                   = 3379                
hdr_size                  = 10493456            
l2_compress_zeros         = 0                   
l2_abort_lowmem           = 0                   
l2_io_error               = 0                   
l2_hdr_size               = 0                   
l2_cksum_bad              = 0                   
l2_evict_reading          = 0                   
arc_meta_limit            = 16900368384         
prefetch_metadata_misses  = 3383                
l2_evict_lock_retry       = 0                   
hash_collisions           = 1442                
l2_writes_hdr_miss        = 0                   
evict_l2_ineligible       = 142102528           
l2_write_bytes            = 0                   
data_size                 = 4370445312          
c                         = 4396690358          
l2_asize                  = 0                   
class                     = misc                
mutex_miss                = 9                   
snaptime                  = 1158.10348092       
duplicate_buffers_size    = 0                   
l2_misses                 = 0                   
mru_ghost_hits            = 56                  
demand_metadata_hits      = 29252               
arc_meta_max              = 100285688           
c_min                     = 2146100480          
mru_hits                  = 385963              
demand_data_misses        = 2669                
duplicate_buffers         = 0                   
demand_data_hits          = 1217641             
l2_free_on_write          = 0                   
l2_feeds                  = 0                   
misses                    = 20164               
evict_skip                = 1214                
evict_l2_eligible         = 1491629056          
p                         = 2997814168          
l2_compress_successes     = 0                   
recycle_miss              = 821                 
l2_writes_done            = 0                   
----------

Thu Dec  4 16:16:27 IST 2014

l2_size                   = 0                   
other_size                = 14938800            
c_max                     = 67601473536         
hash_elements             = 39052               
duplicate_reads           = 0                   
prefetch_data_hits        = 537                 
l2_read_bytes             = 0                   
demand_metadata_misses    = 12576               
prefetch_data_misses      = 1540                
hash_chain_max            = 2                   
l2_hits                   = 0                   
l2_compress_failures      = 0                   
mfu_ghost_hits            = 0                   
l2_rw_clash               = 0                   
memory_throttle_count     = 0                   
crtime                    = 4.537754192         
arc_meta_used             = 80821952            
hash_elements_max         = 49670               
size                      = 3828283072          
hits                      = 1264949             
mfu_hits                  = 864148              
prefetch_metadata_hits    = 10297               
hash_chains               = 75                  
evict_l2_cached           = 0                   
l2_writes_sent            = 0                   
l2_writes_error           = 0                   
deleted                   = 10138               
hdr_size                  = 8953872             
l2_compress_zeros         = 0                   
l2_abort_lowmem           = 0                   
l2_io_error               = 0                   
l2_hdr_size               = 0                   
l2_cksum_bad              = 0                   
l2_evict_reading          = 0                   
arc_meta_limit            = 16900368384         
prefetch_metadata_misses  = 3383                
l2_evict_lock_retry       = 0                   
hash_collisions           = 1454                
l2_writes_hdr_miss        = 0                   
evict_l2_ineligible       = 144691200           
l2_write_bytes            = 0                   
data_size                 = 3804390400          
c                         = 3830309700          
l2_asize                  = 0                   
class                     = misc                
mutex_miss                = 12                  
snaptime                  = 1161.109480472      
duplicate_buffers_size    = 0                   
l2_misses                 = 0                   
mru_ghost_hits            = 77                  
demand_metadata_hits      = 29256               
arc_meta_max              = 100285688           
c_min                     = 2146100480          
mru_hits                  = 389967              
demand_data_misses        = 2692                
duplicate_buffers         = 0                   
demand_data_hits          = 1224859             
l2_free_on_write          = 0                   
l2_feeds                  = 0                   
misses                    = 20191               
evict_skip                = 1214                
evict_l2_eligible         = 1951429632          
p                         = 2764099257          
l2_compress_successes     = 0                   
recycle_miss              = 972                 
l2_writes_done            = 0                   
----------

-------------- next part --------------
Thu Dec  4 16:14:02 IST 2014
Page Summary                Pages                MB  %Tot
------------     ----------------  ----------------  ----
Kernel                    5667640             22139   34%
ZFS File Data             1171645              4576    7%
Anon                        26071               101    0%
Exec and libs                1181                 4    0%
Page cache                   4609                18    0%
Free (cachelist)             7723                30    0%
Free (freelist)           9887541             38623   59%

Total                    16766410             65493
Physical                 16766409             65493

Thu Dec  4 16:14:06 IST 2014
Page Summary                Pages                MB  %Tot
------------     ----------------  ----------------  ----
Kernel                    6380424             24923   38%
ZFS File Data             1171645              4576    7%
Anon                        26096               101    0%
Exec and libs                1175                 4    0%
Page cache                   4610                18    0%
Free (cachelist)             7729                30    0%
Free (freelist)           9174731             35838   55%

Total                    16766410             65493
Physical                 16766409             65493

Thu Dec  4 16:14:09 IST 2014
Page Summary                Pages                MB  %Tot
------------     ----------------  ----------------  ----
Kernel                    6524733             25487   39%
ZFS File Data             1184093              4625    7%
Anon                        26096               101    0%
Exec and libs                1181                 4    0%
Page cache                   4612                18    0%
Free (cachelist)             7722                30    0%
Free (freelist)           9017973             35226   54%

Total                    16766410             65493
Physical                 16766409             65493

Thu Dec  4 16:14:12 IST 2014
Page Summary                Pages                MB  %Tot
------------     ----------------  ----------------  ----
Kernel                    7290171             28477   43%
ZFS File Data             1184093              4625    7%
Anon                        26122               102    0%
Exec and libs                1181                 4    0%
Page cache                   4612                18    0%
Free (cachelist)             7723                30    0%
Free (freelist)           8252508             32236   49%

Total                    16766410             65493
Physical                 16766409             65493

Thu Dec  4 16:14:16 IST 2014
Page Summary                Pages                MB  %Tot
------------     ----------------  ----------------  ----
Kernel                    7456599             29127   44%
ZFS File Data             1190237              4649    7%
Anon                        26122               102    0%
Exec and libs                1181                 4    0%
Page cache                   4613                18    0%
Free (cachelist)             7723                30    0%
Free (freelist)           8079935             31562   48%

Total                    16766410             65493
Physical                 16766409             65493

Thu Dec  4 16:14:19 IST 2014
Page Summary                Pages                MB  %Tot
------------     ----------------  ----------------  ----
Kernel                    7480950             29222   45%
ZFS File Data             1513949              5913    9%
Anon                        26178               102    0%
Exec and libs                1180                 4    0%
Page cache                   4615                18    0%
Free (cachelist)             7724                30    0%
Free (freelist)           7731814             30202   46%

Total                    16766410             65493
Physical                 16766409             65493

Thu Dec  4 16:14:25 IST 2014
Page Summary                Pages                MB  %Tot
------------     ----------------  ----------------  ----
Kernel                    7481258             29223   45%
ZFS File Data             1513949              5913    9%
Anon                        26174               102    0%
Exec and libs                1181                 4    0%
Page cache                   4616                18    0%
Free (cachelist)             7723                30    0%
Free (freelist)           7731509             30201   46%

Total                    16766410             65493
Physical                 16766409             65493

Thu Dec  4 16:14:28 IST 2014
Page Summary                Pages                MB  %Tot
------------     ----------------  ----------------  ----
Kernel                    7482405             29228   45%
ZFS File Data             1577181              6160    9%
Anon                        26200               102    0%
Exec and libs                1181                 4    0%
Page cache                   4617                18    0%
Free (cachelist)             7723                30    0%
Free (freelist)           7667103             29949   46%

Total                    16766410             65493
Physical                 16766409             65493

Thu Dec  4 16:14:31 IST 2014
Page Summary                Pages                MB  %Tot
------------     ----------------  ----------------  ----
Kernel                    7482519             29228   45%
ZFS File Data             1577181              6160    9%
Anon                        26226               102    0%
Exec and libs                1181                 4    0%
Page cache                   4618                18    0%
Free (cachelist)             7723                30    0%
Free (freelist)           7666962             29949   46%

Total                    16766410             65493
Physical                 16766409             65493

Thu Dec  4 16:14:35 IST 2014
Page Summary                Pages                MB  %Tot
------------     ----------------  ----------------  ----
Kernel                    7482590             29228   45%
ZFS File Data             1577181              6160    9%
Anon                        26226               102    0%
Exec and libs                1180                 4    0%
Page cache                   4619                18    0%
Free (cachelist)             7723                30    0%
Free (freelist)           7666891             29948   46%

Total                    16766410             65493
Physical                 16766409             65493

Thu Dec  4 16:14:38 IST 2014
Page Summary                Pages                MB  %Tot
------------     ----------------  ----------------  ----
Kernel                    7482751             29229   45%
ZFS File Data             1577181              6160    9%
Anon                        26252               102    0%
Exec and libs                1181                 4    0%
Page cache                   4619                18    0%
Free (cachelist)             7723                30    0%
Free (freelist)           7666703             29948   46%

Total                    16766410             65493
Physical                 16766409             65493

Thu Dec  4 16:14:41 IST 2014
Page Summary                Pages                MB  %Tot
------------     ----------------  ----------------  ----
Kernel                    7482755             29229   45%
ZFS File Data             1577181              6160    9%
Anon                        26252               102    0%
Exec and libs                1181                 4    0%
Page cache                   4622                18    0%
Free (cachelist)             7723                30    0%
Free (freelist)           7666696             29948   46%

Total                    16766410             65493
Physical                 16766409             65493

Thu Dec  4 16:14:44 IST 2014
Page Summary                Pages                MB  %Tot
------------     ----------------  ----------------  ----
Kernel                    7483021             29230   45%
ZFS File Data             1577181              6160    9%
Anon                        26278               102    0%
Exec and libs                1181                 4    0%
Page cache                   4623                18    0%
Free (cachelist)             7723                30    0%
Free (freelist)           7666403             29946   46%

Total                    16766410             65493
Physical                 16766409             65493

Thu Dec  4 16:14:48 IST 2014
Page Summary                Pages                MB  %Tot
------------     ----------------  ----------------  ----
Kernel                    7483581             29232   45%
ZFS File Data             1577181              6160    9%
Anon                        26303               102    0%
Exec and libs                1181                 4    0%
Page cache                   4622                18    0%
Free (cachelist)             7724                30    0%
Free (freelist)           7665818             29944   46%

Total                    16766410             65493
Physical                 16766409             65493

Thu Dec  4 16:14:51 IST 2014
Page Summary                Pages                MB  %Tot
------------     ----------------  ----------------  ----
Kernel                    7645816             29866   46%
ZFS File Data             1577181              6160    9%
Anon                        26328               102    0%
Exec and libs                1181                 4    0%
Page cache                   4624                18    0%
Free (cachelist)             7723                30    0%
Free (freelist)           7503557             29310   45%

Total                    16766410             65493
Physical                 16766409             65493

Thu Dec  4 16:14:55 IST 2014
Page Summary                Pages                MB  %Tot
------------     ----------------  ----------------  ----
Kernel                    7645904             29866   46%
ZFS File Data             1577181              6160    9%
Anon                        26328               102    0%
Exec and libs                1181                 4    0%
Page cache                   4625                18    0%
Free (cachelist)             7723                30    0%
Free (freelist)           7503468             29310   45%

Total                    16766410             65493
Physical                 16766409             65493

Thu Dec  4 16:14:58 IST 2014
Page Summary                Pages                MB  %Tot
------------     ----------------  ----------------  ----
Kernel                    7753278             30286   46%
ZFS File Data             1577181              6160    9%
Anon                        26355               102    0%
Exec and libs                1181                 4    0%
Page cache                   4626                18    0%
Free (cachelist)             7723                30    0%
Free (freelist)           7396066             28890   44%

Total                    16766410             65493
Physical                 16766409             65493

Thu Dec  4 16:15:01 IST 2014
Page Summary                Pages                MB  %Tot
------------     ----------------  ----------------  ----
Kernel                    8085909             31585   48%
ZFS File Data             1577181              6160    9%
Anon                        26380               103    0%
Exec and libs                1179                 4    0%
Page cache                   4627                18    0%
Free (cachelist)             7725                30    0%
Free (freelist)           7063409             27591   42%

Total                    16766410             65493
Physical                 16766409             65493

Thu Dec  4 16:15:04 IST 2014
Page Summary                Pages                MB  %Tot
------------     ----------------  ----------------  ----
Kernel                    8085951             31585   48%
ZFS File Data             1577181              6160    9%
Anon                        26380               103    0%
Exec and libs                1181                 4    0%
Page cache                   4628                18    0%
Free (cachelist)             7723                30    0%
Free (freelist)           7063366             27591   42%

Total                    16766410             65493
Physical                 16766409             65493

Thu Dec  4 16:15:08 IST 2014
Page Summary                Pages                MB  %Tot
------------     ----------------  ----------------  ----
Kernel                    8209025             32066   49%
ZFS File Data             1577181              6160    9%
Anon                        26407               103    0%
Exec and libs                1181                 4    0%
Page cache                   4630                18    0%
Free (cachelist)             7723                30    0%
Free (freelist)           6940263             27110   41%

Total                    16766410             65493
Physical                 16766409             65493

Thu Dec  4 16:15:11 IST 2014
Page Summary                Pages                MB  %Tot
------------     ----------------  ----------------  ----
Kernel                    8763032             34230   52%
ZFS File Data             1577181              6160    9%
Anon                        26432               103    0%
Exec and libs                1181                 4    0%
Page cache                   4630                18    0%
Free (cachelist)             7723                30    0%
Free (freelist)           6386231             24946   38%

Total                    16766410             65493
Physical                 16766409             65493

Thu Dec  4 16:15:14 IST 2014
Page Summary                Pages                MB  %Tot
------------     ----------------  ----------------  ----
Kernel                    9388716             36674   56%
ZFS File Data             1577181              6160    9%
Anon                        26432               103    0%
Exec and libs                1181                 4    0%
Page cache                   4631                18    0%
Free (cachelist)             7724                30    0%
Free (freelist)           5760545             22502   34%

Total                    16766410             65493
Physical                 16766409             65493

Thu Dec  4 16:15:17 IST 2014
Page Summary                Pages                MB  %Tot
------------     ----------------  ----------------  ----
Kernel                    9586189             37446   57%
ZFS File Data             1577181              6160    9%
Anon                        26459               103    0%
Exec and libs                1181                 4    0%
Page cache                   4632                18    0%
Free (cachelist)             7723                30    0%
Free (freelist)           5563045             21730   33%

Total                    16766410             65493
Physical                 16766409             65493

Thu Dec  4 16:15:20 IST 2014
Page Summary                Pages                MB  %Tot
------------     ----------------  ----------------  ----
Kernel                    9967338             38934   59%
ZFS File Data             1577181              6160    9%
Anon                        26459               103    0%
Exec and libs                1181                 4    0%
Page cache                   4633                18    0%
Free (cachelist)             7723                30    0%
Free (freelist)           5181895             20241   31%

Total                    16766410             65493
Physical                 16766409             65493

Thu Dec  4 16:15:24 IST 2014
Page Summary                Pages                MB  %Tot
------------     ----------------  ----------------  ----
Kernel                   10087818             39405   60%
ZFS File Data             1577181              6160    9%
Anon                        26484               103    0%
Exec and libs                1181                 4    0%
Page cache                   4635                18    0%
Free (cachelist)             7723                30    0%
Free (freelist)           5061388             19771   30%

Total                    16766410             65493
Physical                 16766409             65493

Thu Dec  4 16:15:27 IST 2014
Page Summary                Pages                MB  %Tot
------------     ----------------  ----------------  ----
Kernel                   10620046             41484   63%
ZFS File Data             1577181              6160    9%
Anon                        26510               103    0%
Exec and libs                1157                 4    0%
Page cache                   4633                18    0%
Free (cachelist)             7750                30    0%
Free (freelist)           4529133             17691   27%

Total                    16766410             65493
Physical                 16766409             65493

Thu Dec  4 16:15:30 IST 2014
Page Summary                Pages                MB  %Tot
------------     ----------------  ----------------  ----
Kernel                   10728161             41906   64%
ZFS File Data             1577181              6160    9%
Anon                        26510               103    0%
Exec and libs                1181                 4    0%
Page cache                   4637                18    0%
Free (cachelist)             7723                30    0%
Free (freelist)           4421017             17269   26%

Total                    16766410             65493
Physical                 16766409             65493

Thu Dec  4 16:15:34 IST 2014
Page Summary                Pages                MB  %Tot
------------     ----------------  ----------------  ----
Kernel                   11230924             43870   67%
ZFS File Data             1577181              6160    9%
Anon                        26536               103    0%
Exec and libs                1181                 4    0%
Page cache                   4637                18    0%
Free (cachelist)             7723                30    0%
Free (freelist)           3918228             15305   23%

Total                    16766410             65493
Physical                 16766409             65493

Thu Dec  4 16:15:38 IST 2014
Page Summary                Pages                MB  %Tot
------------     ----------------  ----------------  ----
Kernel                   11277074             44051   67%
ZFS File Data             1577981              6163    9%
Anon                        26562               103    0%
Exec and libs                1181                 4    0%
Page cache                   4638                18    0%
Free (cachelist)             7723                30    0%
Free (freelist)           3871251             15122   23%

Total                    16766410             65493
Physical                 16766409             65493

Thu Dec  4 16:15:41 IST 2014
Page Summary                Pages                MB  %Tot
------------     ----------------  ----------------  ----
Kernel                   11779320             46012   70%
ZFS File Data             1577981              6163    9%
Anon                        26562               103    0%
Exec and libs                1180                 4    0%
Page cache                   4639                18    0%
Free (cachelist)             7724                30    0%
Free (freelist)           3369004             13160   20%

Total                    16766410             65493
Physical                 16766409             65493

Thu Dec  4 16:15:44 IST 2014
Page Summary                Pages                MB  %Tot
------------     ----------------  ----------------  ----
Kernel                   11968305             46751   71%
ZFS File Data             1577981              6163    9%
Anon                        26588               103    0%
Exec and libs                1181                 4    0%
Page cache                   4641                18    0%
Free (cachelist)             7723                30    0%
Free (freelist)           3179991             12421   19%

Total                    16766410             65493
Physical                 16766409             65493

Thu Dec  4 16:15:47 IST 2014
Page Summary                Pages                MB  %Tot
------------     ----------------  ----------------  ----
Kernel                   12424008             48531   74%
ZFS File Data             1577981              6163    9%
Anon                        26614               103    0%
Exec and libs                1181                 4    0%
Page cache                   4642                18    0%
Free (cachelist)             7723                30    0%
Free (freelist)           2724261             10641   16%

Total                    16766410             65493
Physical                 16766409             65493

Thu Dec  4 16:15:51 IST 2014
Page Summary                Pages                MB  %Tot
------------     ----------------  ----------------  ----
Kernel                   12708257             49641   76%
ZFS File Data             1577981              6163    9%
Anon                        26614               103    0%
Exec and libs                1181                 4    0%
Page cache                   4643                18    0%
Free (cachelist)             7723                30    0%
Free (freelist)           2440011              9531   15%

Total                    16766410             65493
Physical                 16766409             65493

Thu Dec  4 16:15:54 IST 2014
Page Summary                Pages                MB  %Tot
------------     ----------------  ----------------  ----
Kernel                   12960558             50627   77%
ZFS File Data             1577981              6163    9%
Anon                        26640               104    0%
Exec and libs                1181                 4    0%
Page cache                   4644                18    0%
Free (cachelist)             7723                30    0%
Free (freelist)           2187683              8545   13%

Total                    16766410             65493
Physical                 16766409             65493

Thu Dec  4 16:15:58 IST 2014
Page Summary                Pages                MB  %Tot
------------     ----------------  ----------------  ----
Kernel                   13441434             52505   80%
ZFS File Data             1577981              6163    9%
Anon                        26664               104    0%
Exec and libs                1181                 4    0%
Page cache                   4645                18    0%
Free (cachelist)             7723                30    0%
Free (freelist)           1706782              6667   10%

Total                    16766410             65493
Physical                 16766409             65493

Thu Dec  4 16:16:01 IST 2014
Page Summary                Pages                MB  %Tot
------------     ----------------  ----------------  ----
Kernel                   13724075             53609   82%
ZFS File Data             1577981              6163    9%
Anon                        26691               104    0%
Exec and libs                1181                 4    0%
Page cache                   4646                18    0%
Free (cachelist)             7723                30    0%
Free (freelist)           1424113              5562    8%

Total                    16766410             65493
Physical                 16766409             65493

Thu Dec  4 16:16:05 IST 2014
Page Summary                Pages                MB  %Tot
------------     ----------------  ----------------  ----
Kernel                   14075430             54982   84%
ZFS File Data             1577981              6163    9%
Anon                        26691               104    0%
Exec and libs                1181                 4    0%
Page cache                   4647                18    0%
Free (cachelist)             7723                30    0%
Free (freelist)           1072757              4190    6%

Total                    16766410             65493
Physical                 16766409             65493

Thu Dec  4 16:16:08 IST 2014
Page Summary                Pages                MB  %Tot
------------     ----------------  ----------------  ----
Kernel                   14302316             55868   85%
ZFS File Data             1577981              6163    9%
Anon                        26723               104    0%
Exec and libs                1181                 4    0%
Page cache                   4649                18    0%
Free (cachelist)             7723                30    0%
Free (freelist)            845837              3304    5%

Total                    16766410             65493
Physical                 16766409             65493

Thu Dec  4 16:16:12 IST 2014
Page Summary                Pages                MB  %Tot
------------     ----------------  ----------------  ----
Kernel                   14585839             56975   87%
ZFS File Data             1541308              6020    9%
Anon                        26743               104    0%
Exec and libs                1181                 4    0%
Page cache                   4649                18    0%
Free (cachelist)             7724                30    0%
Free (freelist)            598966              2339    4%

Total                    16766410             65493
Physical                 16766409             65493

Thu Dec  4 16:16:16 IST 2014
Page Summary                Pages                MB  %Tot
------------     ----------------  ----------------  ----
Kernel                   14682403             57353   88%
ZFS File Data             1408152              5500    8%
Anon                        26768               104    0%
Exec and libs                1181                 4    0%
Page cache                   4651                18    0%
Free (cachelist)             7723                30    0%
Free (freelist)            635532              2482    4%

Total                    16766410             65493
Physical                 16766409             65493

Thu Dec  4 16:16:19 IST 2014
Page Summary                Pages                MB  %Tot
------------     ----------------  ----------------  ----
Kernel                   15056031             58812   90%
ZFS File Data             1163626              4545    7%
Anon                        26778               104    0%
Exec and libs                1181                 4    0%
Page cache                   4652                18    0%
Free (cachelist)             7723                30    0%
Free (freelist)            506419              1978    3%

Total                    16766410             65493
Physical                 16766409             65493

Thu Dec  4 16:16:23 IST 2014
Page Summary                Pages                MB  %Tot
------------     ----------------  ----------------  ----
Kernel                   15285492             59708   91%
ZFS File Data             1008029              3937    6%
Anon                        26795               104    0%
Exec and libs                1181                 4    0%
Page cache                   4653                18    0%
Free (cachelist)             7429                29    0%
Free (freelist)            432831              1690    3%

Total                    16766410             65493
Physical                 16766409             65493

Thu Dec  4 16:16:27 IST 2014
Page Summary                Pages                MB  %Tot
------------     ----------------  ----------------  ----
Kernel                   15705417             61349   94%
ZFS File Data              857418              3349    5%
Anon                        21468                83    0%
Exec and libs                  73                 0    0%
Page cache                    721                 2    0%
Free (cachelist)            14622                57    0%
Free (freelist)            166691               651    1%

Total                    16766410             65493
Physical                 16766409             65493


-------------- next part --------------
last pid:   996;  load avg:  5.95,  3.04,  1.31;  up 0+00:19:43                                                                                                                 16:16:27
59 processes: 58 sleeping, 1 on cpu
CPU states: 30.6% idle,  1.6% user, 67.8% kernel,  0.0% iowait,  0.0% swap
Kernel: 97556 ctxsw, 2440 trap, 448800 intr, 27430 syscall, 1 fork, 1407 flt
Memory: 64G phys mem, 933M free mem, 4096M total swap, 4096M free swap

   PID USERNAME NLWP PRI NICE  SIZE   RES STATE    TIME    CPU COMMAND
   675 daemon     10  60  -20 2904K 1968K sleep    2:28  4.35% nfsd
   607 root        1  59    0 8308K 6668K sleep    0:00  0.01% perl
   631 root        1  59    0   26M   25M cpu/5    0:00  0.01% top
   711 root        1  59    0 5888K 4180K sleep    0:00  0.01% perl
   705 root        1  59    0 4712K 2628K sleep    0:00  0.00% perl
   633 root        1  59    0 7576K 4828K sleep    0:00  0.00% sshd
   712 root        1  59    0 1644K 1040K sleep    0:00  0.00% tee
   622 root        1  59    0 7576K 4828K sleep    0:00  0.00% sshd
   645 root        1  59    0 7576K 4832K sleep    0:00  0.00% sshd
    10 root       14  59    0 7684K 5664K sleep    0:01  0.00% svc.startd
   580 root       27  59    0   23M   16M sleep    0:01  0.00% fmd
   706 root        1  59    0 1644K 1040K sleep    0:00  0.00% tee
    12 root       19  59    0 9704K 8372K sleep    0:04  0.00% svc.configd
   241 root        7  59    0 4456K 3220K sleep    0:00  0.00% devfsadm
   294 root       34  59    0 7480K 4936K sleep    0:00  0.00% nscd
   367 root        1  59    0 1656K  940K sleep    0:00  0.00% utmpd
   658 daemon      2  60  -20 2888K 1912K sleep    0:00  0.00% lockd
   232 root        5  60  -20 2712K 1600K sleep    0:00  0.00% zonestatd
   322 root        4  59    0   11M 8224K sleep    0:00  0.00% hald
   301 root        5  59    0 5232K 4324K sleep    0:00  0.00% picld
   487 root        3  59    0 5272K 3688K sleep    0:00  0.00% inetd
   331 root        2  59    0 6660K 3368K sleep    0:00  0.00% hald-runner
   225 root       17  59    0 6164K 3312K sleep    0:00  0.00% syseventd
   632 root        1  59    0 5804K 3192K sleep    0:00  0.00% sshd
   644 root        1  59    0 5804K 3192K sleep    0:00  0.00% sshd
   612 root        1  59    0 6508K 3180K sleep    0:00  0.00% hald-addon-netw
   614 root        1  59    0 6508K 3180K sleep    0:00  0.00% hald-addon-cpuf
   621 root        1  59    0 5804K 3096K sleep    0:00  0.00% sshd
    49 netadm      3  59    0 4084K 2836K sleep    0:00  0.00% ipmgmtd
   673 root       13  59    0 4588K 2744K sleep    0:00  0.00% mountd



More information about the OmniOS-discuss mailing list