Post on 19-Jan-2015
description
TESTING SUCKS
NOVEMBER 2011
NICHOLAS FITZROY-DALE
TESTING IS LIKE FLOSSING
© 2011 -‐ CONFIDENTIAL
© 2011 -‐ CONFIDENTIAL
© 2011 -‐ CONFIDENTIAL
© 2011 -‐ CONFIDENTIAL
© 2011 -‐ CONFIDENTIAL
© 2011 -‐ CONFIDENTIAL
3 ©2011 Apkudo Inc. www.apkudo.com
61% CRASHES
54% WON’T INSTALL
99%
4 ©2011 Apkudo Inc. www.apkudo.com
“I TEST MY APP!”
99%
5 ©2011 Apkudo Inc. www.apkudo.com
“I TEST MY APP!”
O RLY?
85% MANUAL
6 ©2011 apkudo, inc. Confidential www.apkudo.com
OH NOES
7 ©2011 apkudo, inc. Confidential www.apkudo.com
AD HOC TESTING
Sucks Keeping up with device versions Testing across various devices
AUTOMATED TESTING Tedious JUnit, Robotium Test plan
8 ©2011 apkudo, inc. Confidential www.apkudo.com
MONKEY! A good start. Simulates inputs. Completely automated.
9 ©2011 apkudo, inc. Confidential www.apkudo.com
MONKEY OVERVIEW
Ac$vity Manager Start / stop ac:vi:es Get list of running ac:vi:es
Window Manager Send touch and key events to foreground app
Package Manager Install and uninstall packages Get list of package ac:vi:es
ocean-‐flisch.deviantart.com
10 ©2011 apkudo, inc. Confidential www.apkudo.com
MONKEY OVERVIEW
Ac$vity Manager setActivityController()
Window Manager inject*Event()
Package Manager queryIntentActivities()
11 ©2011 apkudo, inc. Confidential www.apkudo.com
MONKEY SEE MONKEY DO • Monkey injects “events” - what is an event?
– KeyEvent: Hardware buttons and keyboard – MotionEvent: X, Y, and touches – FlipEvent, NetworkEvent, PowerEvent (mostly
legacy)
Security
• Monkey uses standard Android permissions... that no user apps can use!
public static final String SET_ACTIVITY_WATCHER Since: API Level 1 Allows an application to watch and control how activities are started globally in the system. Only for is in debugging (usually the monkey command).
public static final String INJECT_EVENTS Since: API Level 1 Allows an application to inject user events (keys, touch, trackball) into the event stream and deliver them to ANY window. Without this permission, you can only deliver events to windows in your own process. Very few applications should need to use this permission.
12 ©2011 apkudo, inc. Confidential www.apkudo.com
SECURITY
Monkey vs Monkey Runner
• Monkey is for randomized testing, and runs on the device
• Monkey Runner is for scripting your own tests, and runs on a laptop / desktop
• In fact, Monkey Runner uses Monkey internally
MONKEY VS MONKEY RUNNER
Monkey on Angry Birds
adb shell monkey –v -p com.rovio.angrybirds -s 39 --throttle 10 --pct-touch 100 5000
adb shell monkey -p com.rovio.angrybirds 500014 ©2011 apkudo, inc. Confidential www.apkudo.com
THE BIRDS, THEY ARE ANGRY!
What can Monkey find?
• Key and motion events • Crash information if applicable, as traceback
15 ©2011 apkudo, inc. Confidential www.apkudo.com
MONKEY RESULTS :AllowPackage: com.rovio.angrybirds :IncludeCategory: android.intent.category.LAUNCHER :IncludeCategory: android.intent.category.MONKEY :Switch: #Intent;ac:on=android.intent.ac:on.MAIN;category=android.intent.category.LAUNCHER;launchFlags=0x10000000;component=com.rovio.angrybirds/com.rovio.ka3d.App;end // Allowing start of Intent { act=android.intent.ac:on.MAIN cat=[android.intent.category.LAUNCHER] cmp=com.rovio.angrybirds/com.rovio.ka3d.App } in package com.rovio.angrybirds :Sending Pointer ACTION_DOWN x=554.0 y=357.0 :Sending Pointer ACTION_UP x=558.0 y=350.0 // Rejec:ng start of Intent { act=android.intent.ac:on.CALL_BUTTON cmp=com.android.contacts/.DialtactsAc:vity } in package com.android.contacts :Sending Pointer ACTION_DOWN x=696.0 y=280.0 :Sending Pointer ACTION_UP x=709.0 y=271.0 :Sending Pointer ACTION_MOVE x=-‐1.0 y=1.0 :Sending Pointer ACTION_DOWN x=348.0 y=10.0
What can Monkey find?
• Weird UI sequences – Press button 2 before button 1
• Unexpected inputs – E.G. keyboard input during a Toast
• User-interface race conditions • Activity life-cycle errors
16 ©2011 apkudo, inc. Confidential www.apkudo.com
WHAT CAN MONKEY FIND?
Weird UI sequences
• Monkey presses UI controls in an unexpected way
• Example: Comic Rack: selecting “Options” immediately after application start
• Resolution: Always check preconditions in UI callback functions
17 ©2011 apkudo, inc. Confidential www.apkudo.com
WEIRD UI SEQUENCES
Unexpected inputs
• Monkey simulates keypresses that the application doesn’t expect
• Example: multiple keypresses on the same button
• Resolution: Again, be paranoid when entering UI callback functions
18 ©2011 apkudo, inc. Confidential www.apkudo.com
UNEXPECTED INPUTS
Race condi:ons
• Monkey can press UI elements very quickly, triggering race conditions
• Example: I Heart Radio: PlayerController not initialized
• Resolution: limit use of threads. When they are necessary, perform appropriate locking.
• Same applies to networking.
19 ©2011 apkudo, inc. Confidential www.apkudo.com
RACE CONDITIONS
Ac:vity life-‐cycle errors
• Monkey performs unexpected activity life-cycle actions
• Example: Shazam: Unable to pause activity • Resolution: Become tiresomely familiar
with the Android activity life cycle.
ACTIVITY LIFE CYCLE ERRORS
Mul:-‐device Monkey
• Monkey is really good at testing across multiple devices:
• Correct Market permissions – appropriate-architecture native code, eg armv7
• Unexpected hardware properties • “Weird” hardware failures
MULTI-DEVICE MONKEY
Mul:-‐device Monkey
• OpenGL ES version and features • Dependencies on system libraries • Incorrect or missing minSdkVersion
CORRECT MARKET FILTERS
Unexpected hardware proper:es
• The device has an unexpected hardware feature (e.g. a strange screen resolution)
• Examples: RDefense Free, Zombie Killer • Best resolution: Test on multiple devices
UNEXPECTED HARDWARE PROPERTIES
Weird hardware failures
• Not all devices “just work” • Example: PaperCamera doesn’t handle
RuntimeError from takePicture() • Resolution: Test on multiple devices. Be
wary of Java unchecked errors.
24 ©2011 apkudo, inc. Confidential www.apkudo.com
WEIRD HARDWRE FAILURES
Limita:ons
• False positives – “Impossible” sequences of actions / speeds
• Very limited testing strategy – What if most of your app is hidden behind a login
screen? • Some Android pain points not covered
– E.G. screen rotation, layout issues
25 ©2011 apkudo, inc. Confidential www.apkudo.com
MONKEY LIMITATIONS
Apkudo
• “See your app run on every Android device” – Monkey test on 289 phones and tablets – Free – Private beta right now: email
sayhi@apkudo.com if interested
26 ©2011 apkudo, inc. Confidential www.apkudo.com
AND APKUDO
Thank you. NFD@ .COM