MobileBench Consortium 2013
-
Upload
ryo-jin -
Category
Technology
-
view
2.442 -
download
0
Transcript of MobileBench Consortium 2013
Copyright© MobileBench Consortium, 2013
MobileBenchMobileBench ConsortiumConsortium
MobileBench
Standard performance metric for Mobile
2013.07
Copyright© MobileBench Consortium, 2013
Lack of credible Benchmark tools
• The biggest motivation of developing a new benchmark tool was the lack of credibility, fairness, and accuracy among conventional benchmark tools in the market.
No detailed information for device No detailed information for device comparisoncomparison
Small Test items and coverageSmall Test items and coverage Hard Hard to get feedbackto get feedback Difficult to interpret the resultsDifficult to interpret the results Wide fluctuationWide fluctuation User experiences reflected insufficientlyUser experiences reflected insufficiently
1 2 3 4
Copyright© MobileBench Consortium, 2013
Goal for MobileBench consortium
FairnessFairnessEnsure the fairness through opening source-code
ConsistencyConsistencyMinimize the standard deviation of results
+ User Experience+ User Experiencethe performance evaluation based-on user scenario
+ Flexibility+ FlexibilitySupport various evaluation functions for Analysis
DistinctionDistinctionMaximize device performance
Copyright© MobileBench Consortium, 2013
MobileBenchTM Strength (1/5)
Avg: 0.7
Avg: 4.4
Avg: 12.0
Avg: 0.6
ConsistencyMinimize the standard deviation of results
Copyright© MobileBench Consortium, 2013
MobileBenchTM Strength (2/5)
DistinctionMaximize device performance
108%
73%
17%12%
Spec.
Copyright© MobileBench Consortium, 2013
MobileBenchTM Strength (3/5)
FlexibilitySupport various evaluation functions for analysis
Copyright© MobileBench Consortium, 2013
MobileBenchTM Strength (4/5)
MobileBenchMobileBenchTMTM
ResultResult
User Experiencethe performance evaluation based-on user scenario
Other ToolOther ToolResultResult
Difference ofstorage
performance
MobileBenchMobileBenchTMTM --UXUXResultResult
Copyright© MobileBench Consortium, 2013
MobileBenchTM Strength (5/5)
System MonitoringMonitor the behavior of each device in the background
Processor Usage & Freq.
Memory Usage
Storage I/O
MFLOPS/MIPS/Memory/StorageMFLOPS/MIPS/Memory/Storage Java-scriptJava-script Web-loading / HTML5Web-loading / HTML5 UI-stressUI-stress
* Time resolution 1sec * Time resolution 1sec
Copyright© MobileBench Consortium, 2013
MobileBench Suite
MobileBench MobileBench-UX
MobileBenchMobileBench SuiteSuite
Mobile ApplicationMobile Application Device Benchmark on Device Benchmark on PlatformPlatform Device Benchmark on Device Benchmark on KernelKernel Synthetic WorkloadSynthetic Workload System MonitorSystem Monitor
Mobile App. + Mobile App. + PC AgentPC Agent Mobile Device Control via USBMobile Device Control via USB Real Workload/ApplicationReal Workload/Application ScenarioScenario--basedbased System MonitorSystem Monitor
Copyright© MobileBench Consortium, 2013
MobileBench Features
• MobileBench is a total benchmark solution for mobile devices. It can measure the performance of each H/W component as well as web-related and UI performance.
1. System Performance:- CPU/Memory/Storage
2. Web browsing:- Javascript, Web Loading,
HTML5 compatibility
3. User Experience:- U/I Stress, Battery Eater
4. Special Feature:- System Monitor, Email
Reporting
MobileBenchTM Features
Copyright© MobileBench Consortium, 2013
MobileBench Features
Features AP Memory Storage Function in Advanced Mode
Processor Core #, Time control
Memory Thread #, Request size
Storage Chunk size, Pre-condition, Native access
Java Script None
Web Loading None
HTML5 Compatible None
Graphic 3D Graphic (TBD)
MobileBenchTM
Copyright© MobileBench Consortium, 2013
MobileBench-UX Features
• MobileBench-UX is a PC software to measure UX performance controlling mobile devices with PC via USB connection. It can simulate all the different user scenario, changing the combination of scenario items. Since all the test items are what users actually do every day, this tool can actually represent and score the real user experience.
One Click!
UX based bench
Multitasking bench
Multimedia bench
Automatic reporting
CameraVideo Playback
Video Creating
Application Installation
Application Loading
Media Scanning
I/O Operation
Boot Test
Copyright© MobileBench Consortium, 2013
MobileBench-UX Features
Category Item AP Memory Storage Function
Utility
Reboot Rebooting time
Media Scanning Android media scan service time
External File I/O PC to phone file copy time
MTP Test Android media transfer protocol
App. Install Application install time
Multi-media
Video Playback-SMovie frame drop/delay time- Background storage file copy
Video Playback-MMovie frame drop/delay time - Background memory access
Video Creating MPEG-1 video encoding time
Video Trans-coding Movie, MPEG-1 video encoding time
Camera Taking pictures and saving time
Multi-tasking
App. Switching 20-dummy application switching time
Multi-Popup Dual-movie play
Web Browsing Web page loading & zoom in-out
MobileBenchTM-UX
Copyright© MobileBench Consortium, 2013
MobileBench Consortium
• Organization Structure
Board of Director(Chair & Directors)
Special Committees Work Committees
CFO
SecretaryPresident
Tool Maintainer
Board of Director
WorkCommittee
Spec. releaseApproval
Spec.Approval
Specification& Tools
Developer Yes Yes Yes Yes Yes
Adopter No Yes No Yes Yes
Honorary* No Yes No No Yes
Non-member No No No NoNo
(Lite version only)
* Non-Profit organization, educational institutes may be participated as “Honorary” membership with limited right for FREE
• Membership types
Copyright© MobileBench Consortium, 2013
Consortium Roadmap
• First General Assembly is set for 26th of September 2013 at Shenzhen, China• Open for anyone for 1st General Assembly and launching event of
MobileBench suit 2013 for FREE
2013 2014
2Q 3Q 4Q 1Q
Jun.
Consortium
Development
Sep. Mar.
Milestone
2nd Consortium Meeting
Milestone setup for MobileBench 2014
Sep.Jun. Mar.MobileBench 2013 rev.1.0
1st MobileBenchGeneral Assembly
Release of MobileBench suit 2013
1st BoD @ Beijing
Release of beta MobileBench Suit
Copyright© MobileBench Consortium, 2013
MobileBench Consoritium
Register in advance to secure your spot for 1st MobileBench Consoritum General Assembly for FREESeptember 26th 2013, Shenzhen, China(25th for Press & Board of Director meeting)
Hotels and more information will be informed later
Copyright© MobileBench Consortium, 2013
Thank youThank you