[OmniOS-discuss] ARC, L2ARC and metadata questions

[Richard Elling]

On Oct 26, 2014, at 3:03 AM, Henrik Johansson <henrikj at henkis.net> wrote:

> Hi,
> 
> I have a random read-only workload against 50M which are just above 400KB each. I have built the dataset on a set of mirrored(10 mirrors)  devices for best performance but are looking into using a larger ARC or L2ARC to improve performance. I do realise that the dataset is way to large to cache, but caching metadata improves performance and latency for reading.

[I presume you mean quantity 50M of files, each just above 400KB in size]

> 
> In a test on a subset and accesses 1M files with primarycache set to metadata for the dataset, it did improve performance but used 12-18GB of RAM in the ARC (will run it again when possible forgot the exact number). Anyway, it was far to much RAM to be realistic for us to put in each node for metadata caching.

You'll need to look at arc_meta_used, not the (ARC) size. For convenience, the high water
mark of arc_meta_used is stored in arc_meta_max. For example, on a smallish VM I have:
	# kstat -p ::arcstats:arc_meta_*
	zfs:0:arcstats:arc_meta_limit	1202388480
	zfs:0:arcstats:arc_meta_max	48319112
	zfs:0:arcstats:arc_meta_used	48273456

NB, for OmniOS/illumos, arc_meta_limit is not a hard limit -- it is a decision point above which
the eviction algorithm changes priority.

> 
> I have three questions I need help with:
> 
> 1. How do I calculate required ARC metadata size for a dataset, x bytes per block and file size / block size to get the number of blocks?

It is much easier to measure.

> 
> 2. If I where to use a L2ARC for metadata  I need file size / block size * no files * 200 bytes of RAM ( 400 / 128 * 50M * 200 = ~ 37GB? ) to address the larger L2ARC?

This is a good starting point. Recall that this is a cache, so it depends on your working set size.
Again, this is easier to measure the size of the headers used. The "200 bytes" can vary based on
OS release and whether you are using a debug kernel (non-debug kernels are smaller)
	kstat -p ::arcstats:l2_hdr_size

> 
> 3. I have disabled atime and prefetch since it had 100% miss, are there anything else we could do to to improve performance?

Disabling data prefetch doesn't impact metadata prefetch. You can measure both.

DNLC is probably too small, so you should look at the DNLC hit rate, too. Note that when you get
above the arc_meta_limit, DLNC gets reduced, which is probably not what you want. So if your DNLC
hit rate looks poor (< 90%) then consider increasing both the DNLC size and arc_meta_limit.
 -- richard

> 
> Thanks
> Henrik
> _______________________________________________
> OmniOS-discuss mailing list
> OmniOS-discuss at lists.omniti.com
> http://lists.omniti.com/mailman/listinfo/omnios-discuss