CS240Stage3AbstractionsforPracticalSystems

CachingandthememoryhierarchyOperatingsystemsandtheprocessmodelVirtualmemoryDynamicmemoryallocationVictorylap

MemoryHierarchy:Cache

MemoryhierarchyCachebasicsLocalityCacheorganizationCache-awareprogramming

HowdoesexecutiontimegrowwithSIZE?int array[SIZE];fillArrayRandomly(array); int s = 0;

for (int i = 0; i < 200000; i++) {for (int j = 0; j < SIZE; j++) {s += array[j];

}}

4SIZE

TIME

reality

5

0

5

10

15

20

25

30

35

40

45

0 1000 2000 3000 4000 5000 6000 7000 8000 9000

SIZE

Time

Processor-MemoryBottleneck

6

MainMemory

CPU Reg

Processorperformancedoubledaboutevery18months Busbandwidth

evolvedmuchslower

Bandwidth:256bytes/cycleLatency:1-fewcycles

Bandwidth:2Bytes/cycleLatency:100cycles

Solution:caches

Cache

Example

CacheEnglish:n.ahiddenstoragespaceforprovisions,weapons,ortreasuresv.tostoreawayinhidingforfutureuse

ComputerScience:n.acomputermemorywithshortaccesstimeusedtostorefrequentlyorrecentlyusedinstructionsordatav. tostore[data/instructions]temporarilyforlaterquickretrieval

AlsousedmorebroadlyinCS:softwarecaches,filecaches,etc.

7

GeneralCacheMechanics

8

0 1 2 3

4 5 6 7

8 9 10 11

12 13 14 15

8 9 14 3Cache

Memory Larger,slower,cheaper.Partitionedintoblocks (lines).

Dataismovedinblockunits

Smaller,faster,moreexpensive.Storessubsetofmemoryblocks.

(lines)

CPU Block: unitofdataincacheandmemory.(a.k.a.line)

CacheHit

9

0 1 2 3

4 5 6 7

8 9 10 11

12 13 14 15

8 9 14 3Cache

Memory

1.Requestdatainblock b.Request:14

142.Cachehit:

Blockbisincache.

CPU

9

CacheMiss

10

0 1 2 3

4 5 6 7

8 9 10 11

12 13 14 15

8 9 14 3Cache

Memory

1.Request datainblockb.Request:12

2.Cache miss:blockisnot incache

4.Cachefill:Fetchblockfrommemory,storeincache.

Request:12

12

12

9

9

12

3.Cacheeviction:Evictablocktomakeroom,maybestoretomemory.

PlacementPolicy:wheretoputblockincache

ReplacementPolicy:whichblocktoevict

CPU

Locality:whycacheswork

Programstendtousedataandinstructionsataddressesnearorequaltothosetheyhaveusedrecently.

Temporallocality:Recentlyreferenceditemsarelikelytobereferencedagaininthenearfuture.

Spatiallocality:Itemswithnearbyaddressesarelikelytobereferencedclosetogetherintime.

Howdocachesexploittemporalandspatiallocality?

11

block

block

Locality#1

Data:Temporal:sum referencedineachiterationSpatial:arraya[] accessedinstride-1 pattern

Instructions:Temporal:executelooprepeatedlySpatial:executeinstructionsinsequence

Assessinglocalityincodeisanimportantprogrammingskill.

12

sum = 0;for (i = 0; i < n; i++) {

sum += a[i];}return sum;

Whatisstoredinmemory?

Locality#2

13

a[0][0] a[0][1] a[0][2] a[0][3]a[1][0] a[1][1] a[1][2] a[1][3]a[2][0] a[2][1] a[2][2] a[2][3]

1:a[0][0]2:a[0][1]3:a[0][2]4:a[0][3]5:a[1][0]6:a[1][1]7:a[1][2]8:a[1][3]9:a[2][0]10:a[2][1]11:a[2][2]12:a[2][3]

stride1

int sum_array_rows(int a[M][N]) {int sum = 0;

for (int i = 0; i < M; i++) {for (int j = 0; j < N; j++) {

sum += a[i][j];}

}return sum;

}

row-majorMxN2DarrayinC

Locality#3

14

int sum_array_cols(int a[M][N]) {int sum = 0;

for (int j = 0; j < N; j++) {for (int i = 0; i < M; i++) {

sum += a[i][j];}

}return sum;

}

1:a[0][0]2:a[1][0]3:a[2][0]4:a[0][1]5:a[1][1]6:a[2][1]7:a[0][2]8:a[1][2]9:a[2][2]10:a[0][3]11:a[1][3]12:a[2][3]

strideN

row-majorMxN2DarrayinC

…

…a[0][0] a[0][1] a[0][2] a[0][3]a[1][0] a[1][1] a[1][2] a[1][3]a[2][0] a[2][1] a[2][2] a[2][3]

Locality#4

Whatis"wrong"withthiscode?Howcanitbefixed?

15

int sum_array_3d(int a[M][N][N]) {int sum = 0;

for (int i = 0; i < N; i++) {for (int j = 0; j < N; j++) {

for (int k = 0; k < M; k++) {sum += a[k][i][j];

}}

}return sum;

}

CostofCacheMissesMisscostcouldbe100× hitcost.

99%hitscouldbetwiceasgoodas97%.How?Assumecachehittimeof1cycle,misspenaltyof100cycles

Meanaccesstime:97%hits:1cycle+0.03*100cycles=4cycles99%hits:1cycle+0.01*100cycles=2cycles

16

hit/missrates

CachePerformanceMetrics

MissRateFractionofmemoryaccessestodatanotincache(misses/accesses)Typically: 3%- 10%forL1;maybe<1% forL2,dependingonsize,etc.

HitTimeTimetofindanddeliverablockinthecachetotheprocessor.Typically:1- 2clockcyclesforL1;5- 20clockcyclesforL2

MissPenaltyAdditionaltimerequiredoncachemiss=mainmemoryaccesstimeTypically50- 200cyclesforL2 (trend:increasing!)

17

Memory

memoryhierarchywhydoesitwork?

persistentstorage(harddisk, flash,overnetwork,cloud,etc.)

mainmemory(DRAM)

L3cache(SRAM,off-chip)

L1cache(SRAM,on-chip)

L2cache(SRAM,on-chip)

registerssmall,fast,power-hungry,expensive

large,slow,power-efficient,cheap

explicitlyprogram-controlled

Cache Organization:KeyPointsBlockFixed-sizeunitofdata inmemory/cache

PlacementPolicyWhereinthecacheshouldagivenblockbestored?

§ direct-mapped,setassociative

ReplacementPolicyWhatifthereisnoroominthecacheforrequesteddata?

§ leastrecentlyused,mostrecentlyused

WritePolicyWhenshouldwritesupdatelowerlevelsofmemoryhierarchy?

§ writeback,writethrough,writeallocate,nowriteallocate

Blocks 00000000

00001000

00010000

00011000

Memory(byte)address

00010010

Divideaddressspaceintofixed-sizealignedblocks.powerof2

fullbyteaddress

BlockIDaddressbits- offsetbits

offsetwithinblocklog2(blocksize)

Example:blocksize=8

block

0

block

1

block

2

block

3

00010001000100100001001100010100000101010001011000010111

rememberwithinSameBlock?(PointersLab) ...

Note:draw

ingaddressorderdifferentlyfromhereon!

PlacementPolicy

00011011

IndexCache

S=#slots=4

Small,fixednumberofblockslots.

Large,fixednumberofblockslots.

Memory Mapping:index(BlockID)=???BlockID

0000000100100011010001010110011110001001101010111100110111101111

Placement:Direct-Mapped

22

00011011

Index

0000000100100011010001010110011110001001101010111100110111101111

Memory Mapping:index(BlockID)=BlockIDmod SBlockID

Cache

S=#slots=4

(easyforpower-of-2blocksizes...)

Placement:mappingambiguity

23

00011011

Index

0000000100100011010001010110011110001001101010111100110111101111

Memory

Whichblockisinslot2?

BlockID

Cache

S=#slots=4

Mapping:index(BlockID)=BlockIDmod S

Placement:Tagsresolveambiguity

24

00011011

Index

0000000100100011010001010110011110001001101010111100110111101111

Memory

BlockIDbitsnotusedforindex.

BlockID

Tag Data00110101

Cache

S

Mapping:index(BlockID)=BlockIDmod S

Address=Tag,Index,Offset

00010010 fullbyteaddress

BlockIDAddressbits- Offsetbits

Offsetwithinblocklog2(blocksize)=b

#addressbits

BlockIDbits- IndexbitsTag

log2(# cacheslots)Index

a-bitAddresssbits(a-s-b)bits bbits

OffsetTag Index

Wherewithinablock?

Whatslotinthecache?Disambiguatesslotcontents.

Placement:Direct-Mapped

26

00011011

Index

0000000100100011010001010110011110001001101010111100110111101111

Memory

(stilleasyforpower-of-2blocksizes...)

BlockID

Cache

Whynotthismapping?index(BlockID)=BlockID/ S

Apuzzle.

Cachestartsempty.Access(address,hit/miss)stream:

(10,miss),(11,hit),(12,miss)

Whatcouldtheblocksizebe?

27

blocksize>=2bytes blocksize<8bytes

Placement:directmappingconflicts

Whathappenswhenaccessinginrepeatedpattern:0010,0110,0010,0110,0010...?

28

00011011

Index

0000000100100011010001010110011110001001101010111100110111101111

BlockID

cacheconflictEveryaccesssuffersamiss,evictscachelineneededbynextaccess.

Placement:SetAssociative

29

0

1

2

3

Set

2-way4sets,

2blockseach

0

1

Set

4-way2sets,

4blockseach

01234567

Set

1-way8sets,

1blockeach

directmapped

0

Set

8-way1set,

8blocks

fullyassociative

Mapping:index(BlockID)=BlockIDmod S

S=#slotsincachesets

Oneindexperset ofblockslots.Storeblockinany slotwithinset.

Replacementpolicy:ifsetisfull,whatblockshouldbereplaced?Common: leastrecentlyused(LRU)buthardwareusuallyimplements“notmostrecentlyused”

Example:Tag,Index,Offset?

index(1101)=____

4-bitAddress OffsetTag Index

tagbits ____setindexbits ____blockoffsetbits____

Direct-mapped4slots2-byteblocks

Example:Tag,Index,Offset?

16-bitAddress OffsetTag IndexE-wayset-associativeS slots16-byteblocks

01234567

Set

0

1

2

3

Set

0

1

Set

E=1-wayS=8sets

E=2-wayS=4sets

E=4-wayS=2sets

tagbits ____setindexbits ____blockoffsetbits ____index(0x1833) ____

tagbits ____setindexbits ____blockoffsetbits ____index(0x1833) ____

tagbits ____setindexbits ____blockoffsetbits ____index(0x1833) ____

ReplacementPolicyIfsetisfull,whatblockshouldbereplaced?

Common: leastrecentlyused(LRU)(buthardwareusuallyimplements“notmostrecentlyused”

Anotherpuzzle:Cachestartsempty,usesLRU.Access(address,hit/miss)stream

(10,miss);(12,miss);(10,miss)

32

12isnotinthesameblockas10 12’sblockreplaced10’sblock

direct-mappedcacheassociativity ofcache?

GeneralCacheOrganization(S,E,B)

33

Elinesperset(“E-way”)

Ssets

set

block/line

0 1 2 B-1tagv

validbit B =2b bytesofdatapercacheline(thedatablock)

cachecapacity:SxExBdatabytesaddresssize:t+s+baddressbits

Powersof2

CacheRead

34

E=2e linesperset

S=2s sets

0 1 2 B-1tag1

validbitB=2b bytesofdatapercacheline(thedatablock)

tbits sbits bbitsAddressofbyteinmemory:

tag setindex

blockoffset

databeginsatthisoffset

LocatesetbyindexHitifanyblockinset:

is valid;andhasmatchingtag

Getdataatoffsetinblock

CacheRead:Direct-Mapped (E=1)

35

S=2s sets

tbits 0…01 100Addressofint:

0 1 2 7tagv 3 654

0 1 2 7tagv 3 654

0 1 2 7tagv 3 654

0 1 2 7tagv 3 654

findset

Thiscache:• Blocksize:8bytes• Associativity:1blockperset(directmapped)

CacheRead:Direct-Mapped (E=1)

36

tbits 0…01 100Addressofint:

0 1 2 7tagv 3 654

match?:yes=hitvalid?+

blockoffset

tag 7654

int (4Bytes)ishere

Ifnomatch:oldlineisevictedandreplaced

Thiscache:• Blocksize:8bytes• Associativity:1blockperset(directmapped)

Direct-MappedCachePractice

12-bitaddress16lines,4-byteblocksizeDirectmapped

37

11 10 9 8 7 6 5 4 3 2 1 0

03DFC2111167––––03161DF0723610D5

098F6D431324––––03630804020011B2––––0151112311991190B3B2B1B0ValidTagIndex

––––014FD31B7783113E15349604116D

––––012C––––00BB3BDA159312DA––––02D98951003A1248B3B2B1B0ValidTagIndex

0x354

0xA20

Offsetbits?Indexbits?Tagbits?

Example (E=1)

38

int sum_array_rows(double a[16][16]){double sum = 0;

for (int r = 0; r < 16; r++){for (int c = 0; c < 16; c++){

sum += a[r][c];}

}return sum;

}

32bytes=4doubles

Assume:cold(empty)cache3-bitsetindex,5-bitoffset

aa...arrr rcc cc000

int sum_array_cols(double a[16][16]){double sum = 0;

for (int c = 0; c < 16; c++){for (int r = 0; r < 16; r++){

sum += a[r][c];}

}return sum;

}

Localsinregisters.Assume a is aligned such that&a[r][c] is aa...a rrrr cccc 000

0,0 0,1 0,2 0,3

0,4 0,5 0,6 0,7

0,8 0,9 0,a 0,b

0,c 0,d 0,e 0,f

1,0 1,1 1,2 1,3

1,4 1,5 1,6 1,7

1,8 1,9 1,a 1,b

1,c 1,d 1,e 1,f

32bytes=4doubles

4missesperrowofarray4*16=64misses

everyaccessamiss16*16=256misses

0,0 0,1 0,2 0,3

1,0 1,1 1,2 1,3

2,0 2,1 2,2 2,3

3,0 3,1 3,2 3,3

4,0 4,1 4,2 4,3

0,0:aa...a000 000 000000,4:aa...a000 001 000001,0:aa...a000 100 000002,0:aa...a001 000 00000

Example (E=1)

39

int dotprod(int x[8], int y[8]) {int sum = 0;

for (int i = 0; i < 8; i++) {sum += x[i]*y[i];

}return sum;

}

x[0] x[1] x[2] x[3]y[0] y[1] y[2] y[3]x[0] x[1] x[2] x[3]y[0] y[1] y[2] y[3]x[0] x[1] x[2] x[3]

ifxandyaremutuallyaligned,e.g.,0x00,0x80

ifxandyaremutuallyunaligned,e.g.,0x00,0xA0

x[0] x[1] x[2] x[3]

y[0] y[1] y[2] y[3]

x[4] x[5] x[6] x[7]

y[4] y[5] y[6] y[7]

block=16bytes;8setsincacheHowmanyblockoffsetbits?Howmanysetindexbits?

Addressbits:ttt....tsssbbbbB=16=2b: b=4offsetbitsS=8=2s: s=3indexbits

Addressesasbits0x00000000: 000....000000000x00000080: 000....100000000x000000A0: 000....1010000016bytes=4ints

CacheRead:Set-Associative (Example:E=2)

40

tbits 0…01 100Addressofint:

findset

0 1 2 7tagv 3 6540 1 2 7tagv 3 654

0 1 2 7tagv 3 6540 1 2 7tagv 3 654

0 1 2 7tagv 3 6540 1 2 7tagv 3 654

0 1 2 7tagv 3 6540 1 2 7tagv 3 654

Thiscache:• Blocksize:8bytes• Associativity:2blocksperset

0 1 2 7tagv 3 6540 1 2 7tagv 3 654

CacheRead:Set-Associative (Example:E=2)

41

Thiscache:• Blocksize:8bytes• Associativity:2blocksperset

tbits 0…01 100Addressofint:

compareboth

valid?+ match:yes=hit

blockoffset

tag 7654

int (4Bytes)ishere

Ifnomatch:Evictandreplaceonelineinset.

Example (E=2)

43

float dotprod(float x[8], float y[8]) {float sum = 0;

for (int i = 0; i < 8; i++) {sum += x[i]*y[i];

}return sum;

}

x[0] x[1] x[2] x[3] y[0] y[1] y[2] y[3]Ifxandyaligned,e.g.&x[0]=0,&y[0]=128,canstillfitbothbecauseeachsethasspacefortwoblocks/lines

x[4] x[5] x[6] x[7] y[4] y[5] y[6] y[7]4sets

2blocks/linesperset

TypesofCacheMisses

Cold(compulsory)miss

Conflictmiss

Capacitymiss

Whichonescanwemitigate/eliminate?How?

44

WritingtocacheMultiplecopiesofdataexist,mustbekeptinsync.

Write-hitpolicyWrite-through:Write-back:needsadirtybit

Write-misspolicyWrite-allocate:No-write-allocate:

Typicalcaches:Write-back+Write-allocate,usuallyWrite-through+No-write-allocate,occasionally

45

Write-back,write-allocateexample

46

0xCAFECache

Memory

U

0xFACE

0xCAFE

0

T

U

dirtybittag

1. mov $T,%ecx2. mov $U,%edx3. mov $0xFEED,(%ecx)

a. MissonT.

eax = 0xCAFEecx =Tedx =U

Cache/memorynotinvolved

Write-back,write-allocateexample

47

Cache

Memory 0xFACE

0xCAFE

T

U

dirtybit

1. mov $T,%ecx2. mov $U,%edx3. mov $0xFEED,(%ecx)

a. MissonT.b. EvictU(clean:discard).c. FillT(write-allocate).d. WriteTincache(dirty).

4. mov (%edx),%eaxa. MissonU.tag

T 00xFACE0xFEED 1

eax = 0xCAFEecx =Tedx =U

Write-back,write-allocateexample

48

0xCAFECache

Memory

U

0xFACE

0xCAFE

0

T

U

dirtybittag

eax = 0xCAFEecx =Tedx =U

1. mov $T,%ecx2. mov $U,%edx3. mov $0xFEED,(%ecx)

a. MissonT.b. EvictU(clean:discard).c. FillT(write-allocate).d. WriteTincache(dirty).

4. mov (%edx),%eaxa. MissonU.b. EvictT(dirty:writeback).c. FillU.d. Set%eax.

5. DONE.0xFEED

0xCAFE

ExampleMemoryHierarchy

49

Regs

L1d-cache

L1i-cache

L2unifiedcache

Core0

Regs

L1d-cache

L1i-cache

L2unifiedcache

Core3

…

L3unifiedcache(sharedbyallcores)

Mainmemory

Processorpackage

L1i-cacheandd-cache:32KB,8-way,Access:4cycles

L2unifiedcache:256KB,8-way,Access:11cycles

L3unifiedcache:8MB,16-way,Access:30-40cycles

Blocksize:64bytesforallcaches.

slower,butmorelikelytohit

Typicallaptop/desktopprocessor(c.a.201_)

Aside:softwarecachesExamples

Filesystembuffercaches,webbrowsercaches,databasecaches,networkCDNcaches,etc.

SomedesigndifferencesAlmostalwaysfully-associative

Oftenusecomplexreplacementpolicies

Notnecessarilyconstrainedtosingle“block”transfers

50

Cache-FriendlyCodeLocality,locality,locality.Programmercanoptimizeforcacheperformance

DatastructurelayoutDataaccesspatterns

NestedloopsBlocking(seeCSAPP6.5)

Allsystemsfavor“cache-friendlycode”Performanceishardware-specificGenericrulescapturemostadvantages

Keepworkingsetsmall(temporallocality)Usesmallstrides(spatiallocality)Focusoninnerloopcode

51