JSMeter : Characterizing the Behavior of JavaScript Web Applications
description
Transcript of JSMeter : Characterizing the Behavior of JavaScript Web Applications
![Page 1: JSMeter : Characterizing the Behavior of JavaScript Web Applications](https://reader035.fdocuments.us/reader035/viewer/2022062218/568163f3550346895dd577c7/html5/thumbnails/1.jpg)
Paruj RatanaworabhanKasetsart University, Thailand
Ben Livshits and Ben ZornMicrosoft Research, Redmond
JSMeter: Characterizing the Behavior of JavaScript Web Applications
1
in collaboration withDavid Simmons, Corneliu Barsan, and Allen Wirfs-Brock
![Page 2: JSMeter : Characterizing the Behavior of JavaScript Web Applications](https://reader035.fdocuments.us/reader035/viewer/2022062218/568163f3550346895dd577c7/html5/thumbnails/2.jpg)
Background• Who is this guy?
– Current: Principal Researcher, Microsoft – Past: Professor, U of Colorado; PhD, UC Berkeley
• Research in Software Engineering (RiSE)– 30 researchers and developers– Notables
• Code Contracts (Visual Studio 2010)• Static Driver Verifier in Windows• Peli’s Video Blog at MSDN Channel 9
2
![Page 3: JSMeter : Characterizing the Behavior of JavaScript Web Applications](https://reader035.fdocuments.us/reader035/viewer/2022062218/568163f3550346895dd577c7/html5/thumbnails/3.jpg)
Why Measure JavaScript?
• Standardized, de facto language for the web
– Support in every browser, much existing code
• Browser and JavaScript performance is important
– Are current JavaScript benchmarks representative?
– Limited understanding of JavaScript behavior in real sites
• Who cares?
– Users, web application developers, JavaScript engine developers
3
![Page 4: JSMeter : Characterizing the Behavior of JavaScript Web Applications](https://reader035.fdocuments.us/reader035/viewer/2022062218/568163f3550346895dd577c7/html5/thumbnails/4.jpg)
4
ZDNet 29 May 2008
ghacks.net Dec. 2008
BrowserWars!
![Page 5: JSMeter : Characterizing the Behavior of JavaScript Web Applications](https://reader035.fdocuments.us/reader035/viewer/2022062218/568163f3550346895dd577c7/html5/thumbnails/5.jpg)
Artificial Benchmarks versus Real World Sites
5
Maps Maps
7 V8 programs:
• richards• deltablue• crypto• raytrace• earley-
boyer• regexp• splay
8 SunSpider programs:• 3-draytrace• access-nbody• bitops-nsieve• controlflow• crypto-aes• date-xparb• math-cordic• string-
tagcloud
JSMeter 11 real sites:
Goals of JSMeter Project• Instrument JavaScript execution and measure behavior• Compare behavior of JavaScript benchmarks against real sites• Consider how benchmarks can mislead design decisions
![Page 6: JSMeter : Characterizing the Behavior of JavaScript Web Applications](https://reader035.fdocuments.us/reader035/viewer/2022062218/568163f3550346895dd577c7/html5/thumbnails/6.jpg)
6
How We Measured JavaScript
\ie\jscript\*.cpp
Source-level instrumentation
custom jscript.dll
custom trace fileswebsite visits
Offline analyzers
custom trace files0%
10%20%30%40%50%60%70%80%90%
100%
Constant
Other Str Ops
Concat Op
![Page 7: JSMeter : Characterizing the Behavior of JavaScript Web Applications](https://reader035.fdocuments.us/reader035/viewer/2022062218/568163f3550346895dd577c7/html5/thumbnails/7.jpg)
Visiting the Real Sites
• Getting past page load performance• Attempted to use each site in “normal” way:
7
amazon Search a book, add to shopping cart, sign in, and sign outbing Type in a search query and also look for images and newsbingmap Search for a direction from one city to anothercnn Read front page newsebay Search for a notebook, bid, sing in, and sign outeconomist Read front page news, view comments
facebookLog in, visit a friend pages, browse through photos and comments
gmail Sign in, check inbox, delete a mail, and sign outgoogle Type in a search query and also look for images and newsgooglemap Search for a direction from one city to anotherhotmail Sign in, check inbox, delete a mail, and sign out
![Page 8: JSMeter : Characterizing the Behavior of JavaScript Web Applications](https://reader035.fdocuments.us/reader035/viewer/2022062218/568163f3550346895dd577c7/html5/thumbnails/8.jpg)
Understanding JavaScript Behavior
Code
Events
JavaScript
Objects
8
![Page 9: JSMeter : Characterizing the Behavior of JavaScript Web Applications](https://reader035.fdocuments.us/reader035/viewer/2022062218/568163f3550346895dd577c7/html5/thumbnails/9.jpg)
Code Behavior
Code
Events
JavaScript
Objects
9
• Function size
• Instructions/call
• Code locality
• Instruction mix
![Page 10: JSMeter : Characterizing the Behavior of JavaScript Web Applications](https://reader035.fdocuments.us/reader035/viewer/2022062218/568163f3550346895dd577c7/html5/thumbnails/10.jpg)
Total Bytes of JavaScript Source
10
amazo
n
bingmapeb
ay
faceb
ook
hotmail
richard
scry
ptoea
rley
splay
3d-raytr
ace
bitops-n
sieve
crypto-
aes
math-co
rdic
string-ta
gcloud
0
500
1000
1500
2000
2500
Sour
ce s
ize
(kilo
byte
s)
Real Sites V8 SunSpider
Code|Objects|Events
![Page 11: JSMeter : Characterizing the Behavior of JavaScript Web Applications](https://reader035.fdocuments.us/reader035/viewer/2022062218/568163f3550346895dd577c7/html5/thumbnails/11.jpg)
Static Unique Functions Executed
11
amazo
n
bingm
ap ebay
faceb
ook
goog
le
hotm
ail
richa
rdscry
ptoea
rley
splay
3d-ra
ytrace
bitop
s-nsie
ve
crypto
-aes
math-co
rdic
string
-tagcl
oud
0
500
1,000
1,500
2,000
2,500
3,000
3,500
4,000
Uni
que
Func
tion
s Ex
ecut
ed
Real Sites V8 SunSpider
Code|Objects|Events
![Page 12: JSMeter : Characterizing the Behavior of JavaScript Web Applications](https://reader035.fdocuments.us/reader035/viewer/2022062218/568163f3550346895dd577c7/html5/thumbnails/12.jpg)
Bytecodes / Call
12am
azon
bingmapeb
ay
faceb
ook
hotmail
richard
scry
ptoea
rley
splay
3d-raytr
ace
bitops-n
sieve
crypto-
aes
math-co
rdic
string-ta
gcloud
0.00
50.00
100.00
150.00
200.00
250.00
300.00
Byte
code
s / C
all
function(a,b) { var i=0,elem,pos=a.length; if(D.browser.msie) { while(elem=b[i++]) if(elem.nodeType!=8) a[pos++]=elem; } else while(elem=b[i++]) a[pos++]=elem; return a}
Real Sites V8 SunSpider
Code|Objects|Events
![Page 13: JSMeter : Characterizing the Behavior of JavaScript Web Applications](https://reader035.fdocuments.us/reader035/viewer/2022062218/568163f3550346895dd577c7/html5/thumbnails/13.jpg)
Fraction of Code Executed
13
amazo
n
bingmapeb
ay
faceb
ookgoo
gle
hotmail
richard
scry
ptoea
rley
splay
3d-raytr
ace
bitops-n
sieve
crypto-
aes
math-co
rdic
string-ta
gcloud
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
80.00%
90.00%
100.00%Fr
acti
on o
f Co
de E
xecu
ted Most code
not executed
Real Sites V8 SunSpider
Code|Objects|Events
![Page 14: JSMeter : Characterizing the Behavior of JavaScript Web Applications](https://reader035.fdocuments.us/reader035/viewer/2022062218/568163f3550346895dd577c7/html5/thumbnails/14.jpg)
Hot Function Distribution
14
Real Sites V8 Benchmarks
80% of time in < 10 functions80% of time in 100+ functions
80% 80%
Code|Objects|Events
![Page 15: JSMeter : Characterizing the Behavior of JavaScript Web Applications](https://reader035.fdocuments.us/reader035/viewer/2022062218/568163f3550346895dd577c7/html5/thumbnails/15.jpg)
Object Allocation Behavior
Code
Events
JavaScript
Objects
15
• Allocation by types
• Live heap composition
• Lifetime distribution
![Page 16: JSMeter : Characterizing the Behavior of JavaScript Web Applications](https://reader035.fdocuments.us/reader035/viewer/2022062218/568163f3550346895dd577c7/html5/thumbnails/16.jpg)
Total Bytes Allocated
16
amazo
n
bingmapeb
ay
faceb
ook
hotmail
richard
scry
ptoea
rley
splay
3d-raytr
ace
bitops-n
sieve
crypto-
aes
math-co
rdic
string-ta
gcloud
0
5000
10000
15000
20000
25000
30000
35000
40000
45000To
tal h
eap
data
(ki
loby
tes)
Real Sites V8 SunSpider
Code|Objects|Events
![Page 17: JSMeter : Characterizing the Behavior of JavaScript Web Applications](https://reader035.fdocuments.us/reader035/viewer/2022062218/568163f3550346895dd577c7/html5/thumbnails/17.jpg)
Heap Data by Type
17
amazo
n
bingmap
ebay
hotmail
richards
crypto
earle
ysplay
access-
nbody
contro
lflow
date-xp
arb
regexp
-dna
aggregate
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
ObjectsEnumaratorRegExpMathErrorDateNative FuncStringArraysScript FuncHe
ap D
ata
by T
ype
Real Sites V8 SunSpider
Many functionsRest are strings
Few benchmarksallocate much data
Code|Objects|Events
![Page 18: JSMeter : Characterizing the Behavior of JavaScript Web Applications](https://reader035.fdocuments.us/reader035/viewer/2022062218/568163f3550346895dd577c7/html5/thumbnails/18.jpg)
Live Heap Over Time (gmail)
18
Functions growsteadily
Code|Objects|Events
GC reducessize of heap
Objects growsteadily too
![Page 19: JSMeter : Characterizing the Behavior of JavaScript Web Applications](https://reader035.fdocuments.us/reader035/viewer/2022062218/568163f3550346895dd577c7/html5/thumbnails/19.jpg)
Live Heap over Time (ebay)
19
Heaps repeatedly created, discarded
Heap containsmostly functions
Code|Objects|Events
Heap drops to 0 on page load
![Page 20: JSMeter : Characterizing the Behavior of JavaScript Web Applications](https://reader035.fdocuments.us/reader035/viewer/2022062218/568163f3550346895dd577c7/html5/thumbnails/20.jpg)
2 Search Websites, 2 Architectures
Bing Google
Code|Objects|Events
20
You stay on the same page during your entire visitCode loaded onceHeap is bigger
Every transition loads a new pageCode loaded repeatedlyHeap is smaller
![Page 21: JSMeter : Characterizing the Behavior of JavaScript Web Applications](https://reader035.fdocuments.us/reader035/viewer/2022062218/568163f3550346895dd577c7/html5/thumbnails/21.jpg)
Event Handlers in JavaScript
Code
Events
JavaScript
Objects
21
• Number of events• Sizes of handlers
![Page 22: JSMeter : Characterizing the Behavior of JavaScript Web Applications](https://reader035.fdocuments.us/reader035/viewer/2022062218/568163f3550346895dd577c7/html5/thumbnails/22.jpg)
Event-driven Programming Model• Single-threaded, non-preemptive event
handlers• Example handlers: onabort, onclick, etc.• Very different from batch processing of
benchmarks• Handler responsiveness critical to user
experience
22
Code|Objects|Events
![Page 23: JSMeter : Characterizing the Behavior of JavaScript Web Applications](https://reader035.fdocuments.us/reader035/viewer/2022062218/568163f3550346895dd577c7/html5/thumbnails/23.jpg)
Total Events Handled
23
amazo
nbing
bingmap cnn
ebay
econom
ist
faceb
ookgmail
googlem
ap
hotmail
richard
s
deltab
luecry
pto
raytra
ceea
rleyreg
expsp
lay0
1,000
2,000
3,000
4,000
5,000
6,000
7,000To
tal E
vent
s H
andl
ed
Real Sites V8
Almost no events
Code|Objects|Events
![Page 24: JSMeter : Characterizing the Behavior of JavaScript Web Applications](https://reader035.fdocuments.us/reader035/viewer/2022062218/568163f3550346895dd577c7/html5/thumbnails/24.jpg)
Median Bytecodes / Event Handled
24am
azon
bing
bingm
ap cnn ebay
econo
mist
faceb
ook
gmail
goog
le
goog
lemap
hotm
ail0
50
100
150
200
250
300
350
400
450
500M
edia
n By
teco
des
/ Eve
nt H
andl
ed
Code|Objects|Events
506 2137
![Page 25: JSMeter : Characterizing the Behavior of JavaScript Web Applications](https://reader035.fdocuments.us/reader035/viewer/2022062218/568163f3550346895dd577c7/html5/thumbnails/25.jpg)
Sure, this is all good, but…
• Everyone knows benchmarks are unrepresentative
• How much difference does it make, anyway?• Wouldn’t any benchmarks have similar issues?
25
![Page 26: JSMeter : Characterizing the Behavior of JavaScript Web Applications](https://reader035.fdocuments.us/reader035/viewer/2022062218/568163f3550346895dd577c7/html5/thumbnails/26.jpg)
Cold-code Experiment• Observation
– Real web apps have lots of code (much of it cold)– Benchmarks do not
• Question: What happens if the benchmarks have more code?– We added extra, unused to code to 7 SunSpider
benchmarks– We measured the impact on the benchmark
performance
26
![Page 27: JSMeter : Characterizing the Behavior of JavaScript Web Applications](https://reader035.fdocuments.us/reader035/viewer/2022062218/568163f3550346895dd577c7/html5/thumbnails/27.jpg)
Performance Impact of Cold Code
27
3d-raytr
ace
access
-nbody
bitops-n
sieve
controlflow
crypto-ae
s
math-co
rdic
string-t
agcloud
0
100
200
300
400
500
600
700
800
900
0K200K400K800K1M2M
Tim
e (m
sec)
3d-raytr
ace
access
-nbody
bitops-n
sieve
controlflow
crypto-ae
s
math-co
rdic
string-t
agcloud
0
100
200
300
400
500
600
700
800
900
0K200K400K800K1M2M
Tim
e (m
sec)
Chrome3.0.195.38
IE 88.0.601.18865Cold code has
non-uniform impacton execution time
Cold code makesSunSpider on Chromeup to 4.5x slower
![Page 28: JSMeter : Characterizing the Behavior of JavaScript Web Applications](https://reader035.fdocuments.us/reader035/viewer/2022062218/568163f3550346895dd577c7/html5/thumbnails/28.jpg)
Impact of Benchmarks• What gets emphasis
– Making tight loops fast– Optimizing small amounts of code
• Important issues ignored– Garbage collection (especially of strings)– Managing large amounts of code– Optimizing event handling – Considering JavaScript context between page loads
28
![Page 29: JSMeter : Characterizing the Behavior of JavaScript Web Applications](https://reader035.fdocuments.us/reader035/viewer/2022062218/568163f3550346895dd577c7/html5/thumbnails/29.jpg)
Conclusions• JSMeter is an instrumentation framework
– Used to measure and compare JavaScript applications– High-level views of behavior promote understanding
• Benchmarks differ significantly from real sites– Misleads designers, skews implementations
• Next steps– Develop and promote better benchmarks– Design and evaluate better JavaScript runtimes– Promote better performance tools for JavaScript
developers
29
![Page 30: JSMeter : Characterizing the Behavior of JavaScript Web Applications](https://reader035.fdocuments.us/reader035/viewer/2022062218/568163f3550346895dd577c7/html5/thumbnails/30.jpg)
Additional Resources
• Project: http://research.microsoft.com/en-us/projects/jsmeter/ • Video: Project JSMeter: JavaScript Performance Analysis in the Real World
" - MSDN Channel 9 interview with Erik Meier, Ben Livshits, and Ben Zorn
• Paper:– “JSMeter: Comparing the Behavior of JavaScript Benchmarks with Real Web
Applications”, Paruj Ratanaworabhan, Benjamin Livshits and Benjamin G. Zorn, USENIX 2010 Conference on Web Application Development (WebApps’10), June 2010.
• Additional measurements:– "JSMeter: Characterizing Real-World Behavior of JavaScript Programs", Paruj
Ratanaworabhan, Benjamin Livshits, David Simmons, and Benjamin Zorn, MSR-TR-2009-173, December 2009 (49 pages), November 2009.
30
![Page 31: JSMeter : Characterizing the Behavior of JavaScript Web Applications](https://reader035.fdocuments.us/reader035/viewer/2022062218/568163f3550346895dd577c7/html5/thumbnails/31.jpg)
Additional Slides
31
![Page 32: JSMeter : Characterizing the Behavior of JavaScript Web Applications](https://reader035.fdocuments.us/reader035/viewer/2022062218/568163f3550346895dd577c7/html5/thumbnails/32.jpg)
Related Work• JavaScript
– Richards, Lebresne, Burg, and Vitek (PLDI’10)– Draw similar conclusions
• Java– Doufour et al. (OOPSLA’03), Dieckmann and U.
Hölzle (ECOOP’99)• Other languages
– C++: Calder et al. (JPL’95)– Interpreted languages: Romer et al. (ASPLOS’96)
32