CACHING: tuning your golden hammer

By Ian Simpson from Mashery: an Intel Company

What’s being covered (or not) Caching within the JVM Not focusing on external caches like

memcached Not focusing on sharding Both are important, but we only have an

hour

Motivation for this talk

Overall, caching is great! More performance! Greater scalability! When do we have too much of a good

thing? How do we know if/when we’re doing it

wrong?

When is more not better?

Ultimately caching isn’t free (but is usually cheap)

Some caches are not effective (low cache/hit ratio)

Some caches hold on to too much data Some caches cause resource contention If used incorrectly, caches can destroy

performance

Memory: abundant but not free When we cache, we store data in

memory Generally a replacement for fetching

data from (slow) persistent storage Servers have lots of memory! The JVM only handles so much memory

gracefully*

* This varies based on the garbage collection strategy and JVM vendor, which is beyond the scope of this talk

JVM heap

Split between young, tenured, and permanent

Objects start in young gen Most objects in young gen are short lived

and GC’d Objects in young that survive GCs go to

tenured Objects in tenured are long lived (e.g.

cached data)

Heap o’ trouble

Tenured grows as more objects survive GCs cause “stop the world” events, or

pauses GCs on tenured get more costly the

larger it gets Long GC pauses can cause performance

problems Full heap can cause thrashing (lots of

GCs)

Cache structure and data retention Usually caches are maps of keys to

values Under the covers, often Map<K,V> Anything you put in a Map is strongly

referenced Any reachable strong reference can’t be

GC’d Caches need to be reachable to be

useful Caches don’t automagically clean up

stale data

How do we solve heap issues? Don’t cache more than is necessary

Cache/hit ratio should be high (> 80%) Understand how and when data is used Estimate how large your data is or will

be Use background thread to clean stale

data Guava has support for this

Can I just not use the heap?

Yes! You can do off heap caching! No! It’s not particularly straight forward! On 32-bit JVM, 4GB (or 2) – max heap

available On 64-bit JVM, all available memory –

max heap Space is part of the Java process Not managed by the garbage collector

What’s so difficult about off heap? You have to manage your own memory Allocated as a block via

java.nio.ByteBuffer Can also be allocated via

sun.misc.Unsafe You need to know where things are in the

array Serialization into and deserialization

from byte[] Cost to allocate block of memory

Blocking vs non-blocking cache

Blocking vs non-blocking cache Cache miss means data has to be loaded Loading data means blocking threads

requesting it Generally unavoidable unless returning

nothing is OK Cache hit on stale data means reloading Blocking on reloading data can be

optional Return stale data, reload asynchronously

Blocking cache: pros and cons Pros

Stale data is predictably stale Less resource intensive than loading

asynchronously Cons

Can have huge latency on popular keys across threads

Read mutexes can result in serial read performance

Better with many keys and low key contention

Non-blocking cache: pros and cons Pros

Can return stale data and load in the background

No read mutex required! Cons

Requires a thread pool for asynchronous loading

Data is unpredictably stale Better with fewer keys and high key

contention

Which is which? What do I use? EhCache is a blocking cache Guava cache can be either Neither is the best choice in all cases Data characteristics will help you decide Not sure what to use? Talk to your

coworkers!

Example cases

Product catalog with millions of items Blocking

Subset of products on the homepage Non-blocking

Subset of products on sale for limited time Questionable – contention and staleness

both important Split data on TTL, static vs dynamic Refresh price/inventory frequently and

asynchronously

Local vs distributed caching

Local vs distributed caching

Local means each server has its own cache

Distributed means multiple servers share cache

Local is very simple Distributed is very complex Greater detail of distributed is outside

scope

Local: pros and cons

Pros Basically just a decorated Map No serialization overhead Good ones available for free

Cons Less efficient: more cache misses and

memory usage In large clusters, cache/hit can suffer

greatly In clusters, consistency becomes a problem

Distributed: pros and cons

Pros Fewer cache misses in clusters; improves with

size Consistency across cluster Great for splitting up large data sets

Cons Complex: more data management and network

chatter Serialization overhead for remote retrieval Enterprise solutions cost $$$, free ones more

barebones

Which one do I use?

Again, no one perfect solution; depends on data

Local is quick and dirty Distributed is better for large data sets Distributed is usually better for clusters Distributed comes at a cost of

complexity and $$$ Not sure which one? Talk to your

coworkers!

Example cases

CMS with large view templates Distributed (size and consistency)

Display data like greetings, canned messages, etc Local (small, changes infrequently)

Session store for user behavior Questionable – consistency and resource

concerns Not mission critical data, but good to have Local with sticky sessions an option

Getting closer with your data

Questions you should ask

How expensive is it to load? Size: IO/memory bound; electrons only

move so fast Cost: high DB or CPU load means you must

cache Frequency: death by 1000 cuts

How often will I hit cache? Example: static content often means high

cache hit ratio Example: highly diverse data means low

cache hit ratio

Preparation through estimation Estimate how data will grow

Number of new records per month? How much is considered active?

Estimate how diverse data will become Thousands of unique entries? Millions? Subset of data that is more popular?

Work with product owners on expected use of data

And now for a recap!

A recap: fundamentals

Caching is great, but it’s not free Get to know your data Consider how much memory your app

will use Consider how expensive it is to load data Take cache/hit ratios seriously Not everything needs a cache

A recap: choices to make

Non-blocking: great for bounded data Blocking: great for diverse data Non-blocking: uses more resources,

more stale data Blocking: bad when keys have high

contention Local: simple, cheap, but can impact

memory more Distributed: great for big data sets &

large clusters

Getting runtime info

Track cache statistics via JMX Cache hits and misses Total objects in memory

Tools like Nagios and AppDynamics can track JMX

Guava doesn’t have JMX, but it’s easy to add

BUT WHICH ONE DO I USE?!

Ultimately, you’ll have to figure it out … with the help of your coworkers Before you can decide on what to use,

you need to understand the characteristics of your data

Shameless self promotion

Website: http://www.theotherian.com Has several cache and memory related posts

Github Gists: http://gist.github.com/theotherian Lots of code samples (referenced in blog posts)

Interested in these problems? Mashery is hiring! http://mashery.com/careers API management for companies big and small Scalability is core to our business

Obligatory “Questions” slide

Well?