Q2.12: Benchmarking Techniques
-
Upload
linaro -
Category
Technology
-
view
298 -
download
0
description
Transcript of Q2.12: Benchmarking Techniques
![Page 1: Q2.12: Benchmarking Techniques](https://reader034.fdocuments.us/reader034/viewer/2022051609/547a61aab4af9fef158b4abf/html5/thumbnails/1.jpg)
Benchmarking Techniques
Michael Hope <[email protected]>
bzr branch lp:~michaelh1/+junk/benchmarking-techniques r6
![Page 2: Q2.12: Benchmarking Techniques](https://reader034.fdocuments.us/reader034/viewer/2022051609/547a61aab4af9fef158b4abf/html5/thumbnails/2.jpg)
2
Why benchmark?
![Page 3: Q2.12: Benchmarking Techniques](https://reader034.fdocuments.us/reader034/viewer/2022051609/547a61aab4af9fef158b4abf/html5/thumbnails/3.jpg)
3
Issues are
RelevanceAccuracy
Repeatability
![Page 4: Q2.12: Benchmarking Techniques](https://reader034.fdocuments.us/reader034/viewer/2022051609/547a61aab4af9fef158b4abf/html5/thumbnails/4.jpg)
4
Picking relevant benchmarks:
ProfileWorkloadFeatures
We use SPEC 2000 and EEMBC
We'd like shareable benchmarks
![Page 5: Q2.12: Benchmarking Techniques](https://reader034.fdocuments.us/reader034/viewer/2022051609/547a61aab4af9fef158b4abf/html5/thumbnails/5.jpg)
5
Test Platform
Build / test / benchmark via Linux / web / commodity hardware
![Page 6: Q2.12: Benchmarking Techniques](https://reader034.fdocuments.us/reader034/viewer/2022051609/547a61aab4af9fef158b4abf/html5/thumbnails/6.jpg)
6
Measuring
![Page 7: Q2.12: Benchmarking Techniques](https://reader034.fdocuments.us/reader034/viewer/2022051609/547a61aab4af9fef158b4abf/html5/thumbnails/7.jpg)
7
Realtime timers
See 'man clock_gettime'
![Page 8: Q2.12: Benchmarking Techniques](https://reader034.fdocuments.us/reader034/viewer/2022051609/547a61aab4af9fef158b4abf/html5/thumbnails/8.jpg)
8
<comparison of different timers>
See https://wiki.linaro.org/WorkingGroups/ToolChain/Benchmarks/TimerAccuracy
![Page 9: Q2.12: Benchmarking Techniques](https://reader034.fdocuments.us/reader034/viewer/2022051609/547a61aab4af9fef158b4abf/html5/thumbnails/9.jpg)
9
![Page 10: Q2.12: Benchmarking Techniques](https://reader034.fdocuments.us/reader034/viewer/2022051609/547a61aab4af9fef158b4abf/html5/thumbnails/10.jpg)
10
![Page 11: Q2.12: Benchmarking Techniques](https://reader034.fdocuments.us/reader034/viewer/2022051609/547a61aab4af9fef158b4abf/html5/thumbnails/11.jpg)
11
External influences
![Page 12: Q2.12: Benchmarking Techniques](https://reader034.fdocuments.us/reader034/viewer/2022051609/547a61aab4af9fef158b4abf/html5/thumbnails/12.jpg)
12
Other features to be wary of
schedulergovernor
cpuidle, power management, thermal limiting
SMPbugs, like core lockdown
NEON startup
See “Understanding the Linux Kernel” ch10:http://oreilly.com/catalog/linuxkernel/chapter/ch10.html
![Page 13: Q2.12: Benchmarking Techniques](https://reader034.fdocuments.us/reader034/viewer/2022051609/547a61aab4af9fef158b4abf/html5/thumbnails/13.jpg)
13
Putting things together and running
We use:timer built into the app
run five timescollect everything
post process
![Page 14: Q2.12: Benchmarking Techniques](https://reader034.fdocuments.us/reader034/viewer/2022051609/547a61aab4af9fef158b4abf/html5/thumbnails/14.jpg)
14
Statistics
![Page 15: Q2.12: Benchmarking Techniques](https://reader034.fdocuments.us/reader034/viewer/2022051609/547a61aab4af9fef158b4abf/html5/thumbnails/15.jpg)
15
![Page 16: Q2.12: Benchmarking Techniques](https://reader034.fdocuments.us/reader034/viewer/2022051609/547a61aab4af9fef158b4abf/html5/thumbnails/16.jpg)
16
![Page 17: Q2.12: Benchmarking Techniques](https://reader034.fdocuments.us/reader034/viewer/2022051609/547a61aab4af9fef158b4abf/html5/thumbnails/17.jpg)
17
Standard deviationDispersion / coefficient of variance
t-scorest-test
![Page 18: Q2.12: Benchmarking Techniques](https://reader034.fdocuments.us/reader034/viewer/2022051609/547a61aab4af9fef158b4abf/html5/thumbnails/18.jpg)
18
t=X̄ 1− X̄ 2
√ s12
N 1
+s22
N 2
See http://en.wikipedia.org/wiki/Welch%27s_t_test
![Page 19: Q2.12: Benchmarking Techniques](https://reader034.fdocuments.us/reader034/viewer/2022051609/547a61aab4af9fef158b4abf/html5/thumbnails/19.jpg)
19
Compiler Mean Std CVgcc-4.6.2 1.00 134u 134ugcc-linaro-4.6-2011.11 4.25 5035u 1178u
t = 30,300
![Page 20: Q2.12: Benchmarking Techniques](https://reader034.fdocuments.us/reader034/viewer/2022051609/547a61aab4af9fef158b4abf/html5/thumbnails/20.jpg)
21
Variant Mean Std CVPlain 1.00 320u 320uWith SMS 1.01 430u 426u
t = 118
![Page 21: Q2.12: Benchmarking Techniques](https://reader034.fdocuments.us/reader034/viewer/2022051609/547a61aab4af9fef158b4abf/html5/thumbnails/21.jpg)
22
Variant Mean Std CVPlain 1.00 320u 320uWith vectoriser 1.001 404u 404u
t = 8.27 - significant
![Page 22: Q2.12: Benchmarking Techniques](https://reader034.fdocuments.us/reader034/viewer/2022051609/547a61aab4af9fef158b4abf/html5/thumbnails/22.jpg)
23
Our tools
perfdifftestbetterstabulatePython
LibreOffice!
Other statistical tools like scipy.stat, R, Judge, and ministat
![Page 23: Q2.12: Benchmarking Techniques](https://reader034.fdocuments.us/reader034/viewer/2022051609/547a61aab4af9fef158b4abf/html5/thumbnails/23.jpg)
24http://help.libreoffice.org/Calc/Applying_AutoFilter
![Page 24: Q2.12: Benchmarking Techniques](https://reader034.fdocuments.us/reader034/viewer/2022051609/547a61aab4af9fef158b4abf/html5/thumbnails/24.jpg)
25
www.linaro.org / wiki.linaro.org
people.linaro.org/~michaelh/presentations