[OmniOS-discuss] Strange ARC reads numbers

[Richard Elling]

Hi Filip,

On May 5, 2014, at 3:32 AM, Filip Marvan <filip.marvan at aira.cz> wrote:

> Hello,
>  
> I have storage server with OmniOS LTS and 64 GB RAM. This server was installed one year ago. There are about 50 ZVOLs on that server and they are shared through Comstar iSCSI to KVM servers, which are using these ZVOLs for virtual servers.
>  
> After installation, I created a simple Munin script for monitoring ARC and L2ARC, which is using arcstat.pl script from Mike Harsch (http://blog.harschsystems.com/2010/09/08/arcstat-pl-updated-for-l2arc-statistics/).
>  
> Last week, there was realy huge drop in ARC read statistics (you can see that in attachment). On that graph you can see Total ARC accesses per second. The only thing, that we done in that time, was deletion of about 5 unused ZVOLs with many snapshots and some clones. There was no change in virtual servers load.
>  
> Is there anyone who have any idea, why deleting some data (ZVOLs and snapshots) have so dramatic efect on ARC accesses? There was more that 60% of free space on that ZFS pool.

There are two primary reasons for reduction in the number of ARC reads.
	1. the workload isn't reading as much as it used to
	2. the latency of reads has increased
	3. your measurement is b0rken
there are three reasons...

The data you shared clearly shows reduction in reads, but doesn't contain the answers
to the cause. Usually, if #2 is the case, then the phone will be ringing with angry customers
on the other end.

If the above 3 are not the case, then perhaps it is something more subtle. The arcstat reads
does not record the size of the read. To get the read size for zvols is a little tricky, you can
infer it from the pool statistics in iostat. The subtleness here is that if the volblocksize is 
different between the old and new zvols, then the number of (block) reads will be different
for the same workload.
 -- richard


>  
> Thank you!
> Filip Marvan
> <arc_reads.png>_______________________________________________
> OmniOS-discuss mailing list
> OmniOS-discuss at lists.omniti.com
> http://lists.omniti.com/mailman/listinfo/omnios-discuss

--

Richard.Elling at RichardElling.com
+1-760-896-4422



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://omniosce.org/ml-archive/attachments/20140506/ca55b7f8/attachment-0001.html>