Reducing Garbage Collector Cache Misses Shachar Rubinstein Garbage Collection Seminar.

Reducing Garbage Collector Cache Misses

Shachar Rubinstein

Garbage Collection Seminar

The End!

The general problem

CPU’s are getting fast faster and faster Main memory speed lags behind Result: The cost to access main memory is

increasing

Solutions

Hardware and software techniques:– Memory hierarchy– Prefetcing– Multithreading– Non-blocking caches– Dynamic instruction scheduling– Speculative execution

Great Solutions?

Complex hardware and compilers Ineffective for many programs Attack the manifestation (= memory latency)

and not the source (=poor reference locality)

Not exactly…

Previous work

Improving cache locality in dense matrices using loop transformation

Other profile-driven, compiler directed approach

The GC problem

Little temporal locality. Each live object is usually read only once

during mark phase. Most reads are likely to miss. The new contents are unlikely to be used

more than once.

The GC problem – cont.

The sweep phase, like the mark phase, also touches each object once

That’s since the free list pointers are maintained in the objects themselves

Unlike the mark phase, the sweep phase is more sequential

The GC problem – cont.

The sweep is less likely to use cache contents left by the marker

The allocator is likely to miss again, when the object is allocated

The GC problem - previous work

Older work concentrated on paging performance.

Memory size increase lead to abandoning this goal.

But memory size also lead to huge cache miss penalties.

The largest cache size < heap size This problem is unavoidable.

Previous work

Reducing sweep time for a nearly empty heap

Compiler-based prefetching for recursive data structures

How am I going to improve the situation?

Do some magic! Well no… Use real-time information to improve program

cache locality. The mark and sweep phases offers

invaluable opportunities for improvements– Bring objects earlier to the cache– Reuse freed objects for reallocation

Some numbers

Relative to copy GC– Cache miss rates reduced by 21-42%– Program performance improved by 14-37%

Relative to a page level GC– Cache miss rates reduced by 20-41%– Program performance improved by 18-31%

Road map

Cache conscious data placement using generational GC

– Overview– Short generational GC reminder– Real-time data profiling– Object affinity graph– Combining the affinity graph with GC– Experimental evaluation

Other methods and their experimental results

Overview

A program is instrumented to profile its access patterns

The data is used in the same execution and not the next one.

The data -> affinity graph A new copy algorithm uses the graph to

layout the data while copying.

Generational GC – A reminder

The heap is divided to generations GC activity concentrates on young objects,

which typically die faster. Objects that survive one or more scavenges

are moved to the next generation

Implementation notes

The authors used the UM GC toolkit The toolkit has several steps per generation The authors used a single step for each

generation for simplicity. Each step consists of fixed size blocks The blocks are not necessarily contiguous in

memory

Implementation notes - steps

Implementation notes - steps

The steps are used to encode the objects’ age

An object which survives a scavenge is moved to the next step

Implementation notes – moving between generations

The scavenger collects a generation g and all its younger generations

It starts from objects that are:– In g– Reachable from the roots.

Moving an object is copying it into a TO space.

The FROM space can be reused

Copying algorithm – a reminder

Cheney’s algorithm TO and FROM spaces are

switched Starts from the root set Objects are traversed

breadth-first using a queue Objects are copied to TO

space Terminates when the

queue is empty

Copying algorithm – the queue trick

The algorithm

Did you get it?

Real time data profiling

Earlier program run profile is not good enough

Real time data eliminates:– Profile execution run– Finding inputs

But the overhead must be low!

Great!

Profiling data access patterns

Trace every load and store to heap

Huge overhead (factor of 10!)

Reducing overhead

Use object oriented programs properties

1. Most objects are small, often less than 32 bytes

– No need to distinguish between fields, since cache blocks are bigger

Reducing overhead – cont.

2. Most object accesses are not lightweight– Profiling instrumentation will not incur large

overhead

Don’t believe? Stay awake

Collecting profiling data

“Load”s of base object addresses Uses a modified compiler The compiler retains object type information

for selective loads

Code instrumentation

Collecting profiling data - cont

The base object address is entered to an object access buffer

Implementation note

Uses a page trap for buffer overflow A trap causes a graph to be built Recommended buffer size: 15000 (60KB)

Affinity?

Main Entry: af·fin·i·ty Pronunciation: &-'fi-n&-tEFunction: nounInflected Form(s): plural -tiesEtymology: Middle English affinite, from Middle French or Latin; Middle French afinité, from Latin affinitas, from affinis bordering on, related by marriage, from ad- + finis end, borderDate: 14th century1 : relationship by marriage2 a : sympathy marked by community of interest : KINSHIP b (1) : an attraction to or liking for something <people with an affinity to darkness -- Mark Twain> <pork and fennel have a natural affinity for each other -- Abby Mandel> (2) : an attractive force between substances or particles that causes them to enter into and remain in chemical combination c : a person especially of the opposite sex having a particular attraction for one3 a : likeness based on relationship or causal connection <found an affinity between the teller of a tale and the craftsman -- Mary McCarthy> <this investigation, with affinities to a case history, a psychoanalysis, a detective story -- Oliver Sacks> b : a relation between biological groups involving resemblance in structural plan and indicating a common origin

The object affinity graph

The object affinity graph

Nodes – objects Edges – Temporal affinity between objects An undirected graph

Building the graph

Inserting an object to the queue

Incrementing edges’ weight

All clear?

Demonstration

A

B

A

C

D

D

A

Locality queueObject access buffer Graph

Queue tail

Queue tail

Demonstration

A

B

A

C

D

D

A


A

Queue tail

Queue tail

Demonstration

A

B

A

C

D

D

A


Queue tail

A

B

1

Queue tail

Demonstration

A

B

C

D

D

A


Queue tail

A

B

12

Queue tail

Demonstration

A

B

C

D

D

A


Queue tail

A

B

2

C1

1

Queue tail

Demonstration

A

C

D

D

A


Queue tail

A

B

2

C1

1D

11

Queue tail

Demonstration

A

C

D

A


Queue tail

A

B

2

C1

1D

11

Queue tail

Demonstration

A

C

D

A


Queue tail

A

B

2

C1

1D

22

Queue tail

Demonstration

C

D

A


Queue tail

A

B

2

C1

1D

22

Queue tail

Demonstration

C

D

A


Queue tail

A

B

2

C2

1D

23

Queue tail


A separate affinity graph is built for each generation, except the first.

It uses the fact that the object generation is encoded in its address.

This method prevents placing objects from different generations in the same cache block. (Explanations later on)

Implementation notes – queue size

The locality queue size is important Too small -> Miss temporal relationships Too big -> huge graph, long processing time Recommended: 3.


Re-create or update the graph? Depends on the application

– Access phases should re-create– Uniform behavior should update

In this article – re-create before each scavange

Stop!

Our goal: Produce a cache conscious data layout, so that objects are likely to reside in the same cache block

In English: place objects with high temporal affinity next to each other.

The method: Use the profiling information we’ve collected in the copying process.

GC + Real-time profiling

Use the object affinity graph in the Copying algorithm.

Example – object affinity graph

Example – before step 1

Step 1 – using the graph

Flip roles (TO and FROM) Initialize free and unprocessed to the

beginning of the TO space. Pick a node that is in:

– The root set– and– the affinity graph and has the highest edge weight

Perform a greedy DFS on the graph

Step 1 – cont.

Copy each visited object to the TO space Increment the free pointer Store a forwarding address in the FROM

space

Example – After step 1

Step 2 – continues Cheney’s way

Process all objects between the unprocessed and the free pointers, as in Cheney’s algorithm

Step 3 - cleanup

Ensure all roots are in the TO space If not, process them using Cheney’s

algorithm


The object access buffer can be used as a stack for the DFS

Inaccurate results(?)

The object affinity graph may retain objects not reachable = garbage

They will be incorrectly promoted at most once

Efforts are focused on longer lived objects and not on the youngest generation

Experimental evaluation

Methodology – If we have the time Object oriented programs manipulate small

objects Real-time data profiling overhead The algorithm impact on performance

Size of heap objects

But that’s not the point!

Small objects often die fast

Surviving heap objects

Real-time data profiling overhead

Overall execution time

Overall execution time - notes

No impact on L1 cache because its blocks are 16B

Compared to WLM algorithm

Comparison notes

WLM (Wilson-Lam-Moher) improves program’s virtual memory locality.

It performed worse or close to Cheney’s because of the 2GB memory

What else?

Other methods

Two methods that can be used with the previous one– Prefetch on grey– Lazy sweeping

Assumptions

Non moving mark-sweep collector For simplicity, the collector segregates

objects by size. Each block contains objects of a single size

The collector’s data structure are outside the user-visible heap

A mark bit is reserved for each word in the block

Advantages of “outside the heap” data

The mark phase does not need to examine (=bring to the cache) pointer-free objects

Sequences of small unreachable objects can be reclaimed as a group– A single instruction is needed to examine their

sequence of mark bits– It is used when a heap block turns out to be

empty

The mark phase – a reminder

Ensure that all objects are white. Grey all objects pointed to by a root. while there is a grey object g

– blacken g– For each pointer p in g

if p points to a white object– grey that object.

The mark phase – colors

1 mark bit– 0 is white– 1 is grey/black

Stack– In the stack – grey– Removed from stack - black

The mark GC problem

A significant fraction of time is spent to retrieve the first pointer p from each grey object

About third of the marker’s execution time is spent

This time is expected to increase on future machines

Prefetching

A modern CPU instruction A program can prefetch data into the cache

for future use

Prefetching – cont.

But object reference must be predicted soon enough

For example, if the object is in main memory, it must be prefetched hundred of cycles before its use

Prefetching instructions are mostly inserted by compiler optimizations

Prefetch on grey

When? Prefetch as soon as p is found likely to be a pointer

What? Prefetch the first cache line of the object

To improve performance

The last pointer to be pushed on the mark stack is prefetched first

It minimizes the cases in which a just grayed object is immediately examined

And to improve more

Prefetch a few cache lines ahead when scanning an object

It helps with large objects It prefetches more objects if it isn’t that large

The sweep GC problem

If (reclaimed memory > cache size)– Objects are likely to be evicted from the cache by

the allocator or mutator

Thus, the allocator will miss again when reusing the reclaimed memory

Lazy sweeping

Originally used to reduce page faults Delay the sweeping for the allocator Pages will be reused instead of evicted from

the cache

A reminder

A mark bit is saved for each word in a cache block.

A mark bit is used only if its word is the beginning of an object

Cache lazy sweeping – the collector

Scans for each block its mark bits If all bits are unmarked, the block is added to

the free blocks’ pool without touching it If some bits are marked, it’s added to a

queue of blocks waiting to be swept There are several queues, one or more for

each object size

Cache lazy sweeping – the allocator

Maps the request to the appropriate object free list

Returns the first object from the list If the list is empty

– It sweeps the queue of the right size for a block with some available objects

Experimental results

Measured on two platforms Second platform is to get some calibration on

architecture variation

Pentium III/500 results

HP PA-8000/180 based results

Results conclusions

Prefetch on grey eliminates a third to almost all cache miss overhead in the marker.

But it is dependent on data structures used in the program

Results conclusions – cont.

Collector performance is determined by the marker

The sweep performance is architecture dependent

Conclusions

Be concerned about cache locality or Have a method that does it for you

Conclusions – cont.

Real-time data profiling is feasible Produce cache conscious data layout using

that information May help reduce the performance gap

between high-level to low-level languages

Conclusions – cont.

Prefetch on grey and lazy sweeping are cheap to implement and should be in future garbage collectors

Bibliography

Using Generational Garbage Collection To Implement Cache-Conscious Data Placement - Trishul M. Chilimbi and James R. Larus

Reducing Garbage Collector Cache Misses - Hans-J. Boehm

Further reading

Look at the articles Garbage collection – algorithms for automatic

dynamic memory management – Richard Jones & Rafael Lins

Further reading – cont.

Cecil – – Craig Chambers. “Object-oriented multi-methods

in Cecil.” In Proceedings ECOOP’92, LNCS 615, Springer-Verlag, pages 33–56, June 1992.

– Craig Chambers. “The Cecil language: Specification and rationale.” University of Washington Seattle, Technical Report TR-93-03-05, Mar. 1993.

Hyperion by Dan Simmons

Items

Large objects Inter-generational objects placement Why explicitly build free lists? Experimental methodology Second experimental methodology

Large objects

Ungar and Jackson : – There’s an advantage from not copying large

objects (>= 256 bytes) with the same age

A large object is never copied Each step has an associated set of large

objects

Large objects – cont.

A large object is linked in a doubly linked list. If it survives a collection, it’s removed from its

list and inserted to the TO space list. No compaction is done on large objects.

Large objects – cont.

Read more in David Ungar and Frank Jackson. “An adaptive tenuring policy for generation scavengers.” ACM Transactions on Programming Languages and Systems, 14(1):1–27, January 1992

Two generations, one cache block

How important is co-location of inter-generation objects?

The way to achieve this is to demote or promote.

Two generations, one cache block – cont.

Intra-generation pointers are not tracked. In order to demote safely, it’s needed to

collect its original generation Result: Long collection time

Two generations, one cache block – cont.

Promote can be done safely– The young generation is being collected and its

pointers updated– Pointers from old to young are tracked

The locality benefit will start only when the old generation is collected

Premature promotion

Why explicitly build free lists?

Allocation is fast Heap scanning for unmarked objects can be

fast using mark bits Little additional space overhead is required

Experimental methodology

Vortex compiler infrastructure Vortex supports GGC only for Cecil Cecil – A dynamically typed, purely object-

oriented language. Used Cecil benchmarks Repeated each experiment 5 times and

reported the average

Cecil benchmarks

Cecil benchmarks – cont.

Compiled at highest (o2) optimization level

The platform

Sun Ultraserver E5000 12 167Mhz UltraSPARC processors 2GB memory – To prevent page faults Solaris 2.5.1

The platform - memory

L1 – 16KB direct-mapped, 16B blocks L2 – 1MB unified direct-mapped, 64B blocks 64 entry iTLB and 64 entry dTLB, fully

associative

The platform – memory costs

L1, data cache hit – 1 cycle L1 miss, L2 hit – 6 cycles L2 miss – additional 64 cycles

Second experimental methodology

Two platforms All benchmarks except one are C programs

Pentium measurements

Dual processor 500Mhz Pentium III (but only one used)

100Mhz bus 512KB L2 cache Physical memory > 300MB (why keep it a secret?),

which prevented paging and allowed the whole executable in memory

RedHat 6.1 Benchmarks compiled using gcc with –O2

RISC measurements

A single PA-8000/180 MHz processor Running HP/UX 11 Single level I and D caches, 1MB each

Benchmarks

Execution time measurements are a five runs average

The division between sweep and mark times is arbitrary

Pentium III prefetcht0 introduced a new overhead, so prefetchnta was used. It was less effective eliminating cache miss, though

The end

Lectured by: Shachar Rubinstein

[email protected]

GC seminarMolley Sagiv

Audience:

You

Thanks:

For your patience

The Powerpoint XP effects

My parents

No animals were harmed during this production (except for annoying mosquitoes)

Thank you for listening! (and staying awake…)

Reducing Garbage Collector Cache Misses Shachar Rubinstein Garbage Collection Seminar.

Documents

Transcript of Reducing Garbage Collector Cache Misses Shachar Rubinstein Garbage Collection Seminar.