Benchmarking
-
Upload
david-evans -
Category
Education
-
view
1.109 -
download
2
description
Transcript of Benchmarking
cs4414 Fall 2013University of Virginia
David Evans
Class 15:Benchmarking
University of Virginia cs4414 2
Plan for Today
• Finish up Internet (from last class)• Midterm Discussion• Benchmarking
17 October 2013
University of Virginia cs4414 3
Measuring NetworksLatency
Time from sending a bit until it arrives seconds (or seconds per geographic distance)
Bandwidth Rate at which can you transmitbits per second
17 October 2013
University of Virginia cs4414 4
$ traceroute -q 1 -w 30 www.busselton.wa.gov.autraceroute to busselton.wa.gov.au (203.41.180.233), 64 hops max, 52 byte packets 1 dd-wrt (192.168.1.1) 11.156 ms 2 c-24-127-51-1.hsd1.va.comcast.net (24.127.51.1) 32.497 ms 3 te-2-3-ur01.charlville.va.richmond.comcast.net (68.85.226.149) 13.971 ms 4 xe-11-2-0-0-sur01.charlville.va.richmond.comcast.net (69.139.165.221) 12.312 ms 5 xe-4-1-2-0-ar02.charlvilleco.va.richmond.comcast.net (69.139.165.65) 12.395 ms 6 pos-1-2-0-0-cr01.ashburn.va.ibone.comcast.net (68.86.91.53) 25.624 ms 7 pos-3-10-0-0-cr01.56marietta.ga.ibone.comcast.net (68.86.86.221) 31.483 ms 8 pos-1-9-0-0-cr01.dallas.tx.ibone.comcast.net (68.86.87.233) 52.515 ms 9 he-0-12-0-0-cr01.losangeles.ca.ibone.comcast.net (68.86.86.117) 83.242 ms10 as4637-cr01.losangeles.ca.ibone.comcast.net (75.149.228.222) 78.134 ms11 i-0-2-0-11.tlot-core01.bi.telstraglobal.net (202.40.149.185) 86.131 ms12 i-0-0-0-0.sydo-core01.bx.telstraglobal.net (202.84.140.5) 287.302 ms13 tengige0-1-0-14.oxf-gw2.sydney.telstra.net (203.50.13.133) 300.060 ms14 bundle-ether2.oxf-gw1.sydney.telstra.net (203.50.6.85) 274.270 ms15 bundle-ether1.ken-core4.sydney.telstra.net (203.50.6.5) 270.694 ms16 bundle-ether10.win-core1.melbourne.telstra.net (203.50.11.13) 275.252 ms17 bundle-ether6.fli-core1.adelaide.telstra.net (203.50.11.90) 405.600 ms18 bundle-ether5.wel-core3.perth.telstra.net (203.50.11.19) 411.510 ms19 gigabitethernet0-1.wel13.perth.telstra.net (203.50.115.151) 406.044 ms20 *
17 October 2013
Does traceroute tell us anything about the bandwidth between here an Busselton, Austrailia?
University of Virginia cs4414 5
What Impacts Bandwidth?
17 October 2013
LAN: Ethernet (97.6% efficient for 12Kb packets)WAN: PPP (99.9% efficient – only 1-2 bytes overhead)
IPv4: up to 99.96% efficient
University of Virginia cs4414 6
Below all this: Physical Layer
17 October 2013
University of Virginia cs4414 7
BandwidthHow much data can you transfer in a given amount of time?
17 October 2013
University of Virginia cs4414 8
Improving Bandwidth
• Faster transmission– Train signalers to move semaphore flags faster– Use something less physically demanding to transmit
• Bigger pipes– Have multiple signalers transmit every other letter at the
same time
• Better encoding– Figure out how to code more than 98 symbols with
semaphore signal– Morse code (1840s)
17 October 2013
University of Virginia cs4414 9
Morse CodeRepresent letters with series ofshort and long electrical pulses
17 October 2013
University of Virginia cs4414 10
Circuit Switching
• Reserve a whole path through the network for the whole message transmission
Paris Toulon
Nantes
LyonBourges
Once you start a transmission,know you will have use of the network until it is finished. But,wastes network resources.
17 October 2013
University of Virginia cs4414 11
Packet Switching
• Use one link at a time
Paris Toulon
Nantes
LyonBourges
Interleave messages – send whenever the next link is free.
17 October 2013
University of Virginia cs4414 12
internetwork
A collection of multiple networks connected together, so messages can be transmitted between nodes on different networks.
17 October 2013
University of Virginia cs4414 13
The First (international) internet
17 October 2013
1800: Edelcrantz links Sweden and Denmark telegraph networks to coordinate defense
University of Virginia cs4414 14
End of First (international) internet
17 October 2013
1801: British attach Copenhagen; Sweden doesn’t help Denmark; network disconnected
University of Virginia cs4414 15
The (capital-I) Internet
Packet Switching: Leonard Kleinrock (UCLA) thinks he did, Donald Davies and Paul Baran, Edelcrantz’s signalling network (1809)Internet Protocol: Vint Cerf, Bob KahnVision, Funding (DARPA): J.C.R. Licklider, Bob Taylor Government: Al Gore
First politician to promote Internet, 1986; act to connect government networks to form “Interagency Network”
17 October 2013
Vint Cerf (in the Rotunda, 2010)
University of Virginia cs4414 16
First Use of the Internet
October 1969: First packets on the ARPANet from UCLA to Stanford. Starts to send "LOGIN", but it crashes on the G.
How impressive is this compared to communications event 3 months earlier?
17 October 2013
University of Virginia cs4414 17
20 July 1969:Live b/w video from the moon, transmitted
live to millions of televisions worldwide
17 October 2013
University of Virginia cs4414 18
Today’s Internet: Bandwidth
Ethernet: up to 100 Mbits/sec
My office: ~50 M bits / sec
UVa Wireless: 57 M bits / sec
Cable modem at home:~ 30 M bits / sec
17 October 2013
University of Virginia cs4414 1917 October 2013
projections
1 Petabyte = 1015 bytes1 Million petabytes = 1 zettabyte(NSA’s Utah facility is storing “Zbytes”?)
University of Virginia cs4414 20
Midterm Discussion
17 October 2013
University of Virginia cs4414 21
Easiest Way to Ace Midterm
17 October 2013
4 people selected #9, but none were bold enough to leave all the other answers blank. (You should only be “especially proud of” your answer to #9 if you are confident enough to leave all other answers blank.)
University of Virginia cs4414 22
How many processes should a web browser create?
17 October 2013
1 process
several process
one per “tab”
several per “tab”
University of Virginia cs4414 23
How many threads should a web browser create?
17 October 2013
1 process
several process
one per “tab”
several per “tab”
University of Virginia cs4414 24
How many processes should a web browser developed in Rust
create?
17 October 2013
1 process
several process
one per “tab”
several per “tab”
University of Virginia cs4414 25
How many tasks should a web browser developed in Rust
create?
17 October 2013
1 process
several process
one per “tab”
several per “tab”
University of Virginia cs4414 26
Your technically unsophisticated (but smart, rich, and curious) uncle complains that his laptop is too noisy since the fan is running all the time. He expects that as someone taking upper-level computing courses at an elite, public university, you should be able to help him. What do you do?
17 October 2013
University of Virginia cs4414 2717 October 2013
Hardware problem
or software problem?
University of Virginia cs4414 2817 October 2013
Hardware problem or
software problem?
University of Virginia cs4414 2917 October 2013
SOSP 2009
University of Virginia cs4414 30
Uncle’s Fan Problems
Could it be caused by deadlock?
17 October 2013
University of Virginia cs4414 31
Fan Removes Heat
• CPU generates heat when it is executing lots of instructions
• At least with my MBP, one thread is not enough: 100% CPU usage – no fan200% CPU usage – light fan800% CPU usage – max fan
17 October 2013
University of Virginia cs4414 32
PerformanceBenchmarking
17 October 2013
University of Virginia cs4414 33
Why Benchmark?
17 October 2013
University of Virginia cs4414 34
Developer Benchmarks
• Find bottlenecks: know what to spend time optimizing
• Measure impact of changes to optimize performance
• Predict what resources you will need to scale service (“healthcare.gov”)
17 October 2013
Goal is a benchmark that represents well the (anticipated) actual usage
University of Virginia cs4414 35
Micro vs. Macro Benchmarks
• Micro Benchmarks– Test one specific operation
17 October 2013
University of Virginia cs4414 36
“Customer” Benchmarks
• Allow fair and accurate comparisons between different solutions
• Standard accepted by all/many vendors
17 October 2013
Goal is a benchmark that represents well the (anticipated) actual usage for “typical” users
University of Virginia cs4414 37
“Cheating” on Benchmarks
17 October 2013
How could you “cheat” on a benchmark?
University of Virginia cs4414 3817 October 2013
1. On the Exynos 5410, Samsung was detecting the presence of certain benchmarks and raising thermal limits (and thus max GPU frequency) in order to gain an edge on those benchmarks, and
2. On both Snapdragon 600 and Exynos 5410 SGS4 platforms, Samsung was detecting the presence of certain benchmarks and automatically driving CPU voltage/frequency to their highest state right away. Also on Snapdragon platforms, all cores are plugged in immediately upon benchmark detect.
University of Virginia cs4414 3917 October 2013
Everyone (except Google and NVidia) is cheating!
Samsung is just better at it than anyone else.
University of Virginia cs4414 40
Is It Possible to Prevent Benchmark “Cheating”?
17 October 2013
University of Virginia cs4414 41
Benchmarking Zhtta
17 October 2013
University of Virginia cs4414 4217 October 2013
http://www.comlore.com/redist_files/httperf-www2007-slides.pdf
University of Virginia cs4414 43
Example
17 October 2013
httperf --hog --server=localhost --port=4414 \ --num-conns=1000 --rate=10
10 requests per second
Should you run httperf on the machine running your server?
University of Virginia cs4414 4417 October 2013
Maximum connect burst length: 1
Total: connections 1000 requests 0 replies 0 test-duration 99.901 s
Connection rate: 10.0 conn/s (99.9 ms/conn, <=1 concurrent connections)Connection time [ms]: min 0.0 avg 0.0 max 0.0 median 0.0 stddev 0.0Connection time [ms]: connect 0.1Connection length [replies/conn]: 0.000
Request rate: 0.0 req/s (0.0 ms/req)Request size [B]: 0.0
Reply rate [replies/s]: min 0.0 avg 0.0 max 0.0 stddev 0.0 (19 samples)Reply time [ms]: response 0.0 transfer 0.0Reply size [B]: header 0.0 content 0.0 footer 0.0 (total 0.0)Reply status: 1xx=0 2xx=0 3xx=0 4xx=0 5xx=0
CPU time [s]: user 27.52 system 72.40 (user 27.6% system 72.5% total 100.0%)Net I/O: 0.0 KB/s (0.0*10^6 bps)
Errors: total 1000 client-timo 0 socket-timo 0 connrefused 1000 connreset 0Errors: fd-unavail 0 addrunavail 0 ftab-full 0 other 0
University of Virginia cs4414 45
Macrobenchmarks
Need a workload that represents typical load
17 October 2013
University of Virginia cs4414 4617 October 2013
http://ita.ee.lbl.gov/html/traces.html
University of Virginia cs4414 4717 October 2013
http://www.hpl.hp.com/techreports/1999/HPL-1999-35R1.pdf
University of Virginia cs4414 4817 October 2013
University of Virginia cs4414 4917 October 2013
1998Image from http://www.hpl.hp.com/techreports/1999/HPL-1999-35R1.pdf
University of Virginia cs4414 50
Macrobenchmarks
Need a workload that represents typical load
17 October 2013
Synthetic Workload Replayed Workload
University of Virginia cs4414 51
Weilin’s Test Workload
17 October 2013
> dd if=/dev/urandom of=5K.bin bs=5K count=1> dd if=/dev/urandom of=5M.bin bs=5M count=1> dd if=/dev/urandom of=10M.bin bs=10M count=1> dd if=/dev/urandom of=20M.bin bs=20M count=1> dd if=/dev/urandom of=40M.bin bs=40M count=1> dd if=/dev/urandom of=80M.bin bs=80M count=1> dd if=/dev/urandom of=512M.bin bs=512M count=1
http://www.cs.virginia.edu/~wx4ed/cs4414/ps3/zhtta-test-urls.txt
/5K.bin/index.htm/5K.bin/5K.bin/80M.bin/5K.bin/5K.bin/5K.bin/5K.bin/index.htm/5K.bin…
University of Virginia cs4414 52
Testing Zhtta
17 October 2013
> httperf --server localhost --port 4414 --rate 60 \ --num-conns 60 --wlog=y,./zhtta-test-urls.httperfTotal: connections 60 requests 60 replies 60 test-duration 32.588 s
Connection rate: 1.8 conn/s (543.1 ms/conn, <=46 concurrent connections)Connection time [ms]: min 0.4 avg 1237.8 max 31954.9 median 563.5 stddev 4132.5Connection time [ms]: connect 0.0Connection length [replies/conn]: 1.000
Request rate: 1.8 req/s (543.1 ms/req)Request size [B]: 67.0
Reply rate [replies/s]: min 0.0 avg 2.0 max 11.6 stddev 4.7 (6 samples)Reply time [ms]: response 1215.7 transfer 22.1Reply size [B]: header 71.0 content 16640936.0 footer 0.0 (total 16641007.0)Reply status: 1xx=0 2xx=60 3xx=0 4xx=0 5xx=0
CPU time [s]: user 7.40 system 25.12 (user 22.7% system 77.1% total 99.8%)Net I/O: 29920.6 KB/s (245.1*10^6 bps)
Errors: total 0 client-timo 0 socket-timo 0 connrefused 0 connreset 0Errors: fd-unavail 0 addrunavail 0 ftab-full 0 other 0
University of Virginia cs4414 5317 October 2013
Single-task server
Zhttpto
Zhtta with FIFO
Zhtta with SRPT
Zhtta with SRPT and caching
0 5 10 15 20 25 30 35 40
Test Total Duration (seconds)
Is our memory caching helping?
University of Virginia cs4414 5417 October 2013
Single-task server
Zhttpto
Zhtta with FIFO
Zhtta with SRPT
Zhtta with SRPT and caching
0 5000 10000 15000
Average Response Time (ms)
Single-task server
Zhttpto
Zhtta with FIFO
Zhtta with SRPT
Zhtta with SRPT and caching
0 5 10 15 20 25 30 35 40
Test Duration (seconds)
How good is this?
University of Virginia cs4414 55
Possible to do Much Better!
17 October 2013
Single-task server
Zhttpto
Zhtta with FIFO
Zhtta with SRPT
Zhtta with SRPT and caching
Apache2
0 5 10 15 20 25 30 35 40
Test Duration (seconds)
University of Virginia cs4414 5617 October 2013
Single-task server
Zhttpto
Zhtta with FIFO
Zhtta with SRPT
Zhtta with SRPT and caching
Apache2
0 2000 4000 6000 8000 10000 12000 14000 16000
Average Response Time (ms)
0.2 ms
httperf is measuring “Response Time” as latency to receive first byte!
University of Virginia cs4414 57
Charge
• You should definitely be able to show improvements using Weilin’s benchmark (posted with today’s notes)
• To prevent overly-targeted optimizing, we will also test your servers with a different workload
• Demo will separately check correctness– e.g., if a served file changes, should serve new file
not cached old one (within 10 seconds)– To prevent “cheating”, we may look at your code
17 October 2013
PS3 is due less than one week from today! Still, no one has claimed OpenPiazza!