VanarSena: Automated App Testing

App Testing• Test the app for

– performance problems – crashes

• Testing app in the cloud– Upload app to a service– App is automatically executed in the cloud– Emulating different inputs and conditions– Feedback to the developer pinpointing

performance problems and crashes

VanarSena• Automated Monkey

– Automatically navigate the app in the emulator– Maximize coverage and speed– Emulate impatient users

• Record and Replay– Record app interactions and execution– Transaction-level replay of traces– Control external inputs

• Emulation Framework– Emulate different network conditions – Newt– GPS and sensor inputs – Instrumentation– Device configuration – Emulator and Instrumentation

• Performance and Crash Analysis– Differential performance– Drill down, Critical path, Exception path analysis

Monkey

PhoneEmulator

App Binary

Instrumenter

RunnerInstrumentedApp Binary

FeedbackNavigate

Maximize Coverage

• Hit Test– Find Invokable controls– Turn-off execution of event handlers in code – Hit controls and monitor handlers being called

• Static Analysis– Select controls that maximize static reachability

• Dynamic transaction analysis– Dynamic pick controls with transaction graphs that

maximize coverage

Maximizing Speed• When to interact next?

– Static timeouts can drastically slow down the monkey• Processing Completed Event

– Runtime transaction tracking

ProcessingCompleted

Next InteractionInteraction

Impatient Users• Interact when a transaction is ongoing• On edges slowed down by external inputs

– For e.g. network

Next InteractionInteraction

Record and Replay

• Track user interactions– UI controls, structure

• Monitor execution of the app– Transactions– Record timing of events

• Track external inputs– Network data, Location, – Random number

• Relay user interaction events– Interleaved in time by the execution of transactions– Replay external inputs– Approximate replay

Runner Modes

Automated Runner Replay RunnerHybrid Runner

• Provide traces to go past certain UI controls– For e.g. username/password

• Learn automated runner from many recorded traces

• Newt to emulate various network conditions• Proxy to control and replay network data• Interpose on system APIs to control inputs

– GPS, Sensors etc.• Configure emulator for device configurations• Instrumentation to control

– memory and isolated storage

Emulation Framework

• What inputs matter to the app?• When is performance bad?

• Critical path analysis– When does the critical path change

• Focus development efforts on optimizing the critical path

Performance Analysis

• Tightly control external inputs

– Does including the analytics/ad control affect the performance on other transactions?

– Which JSON library is better?• Given my app data

– Has the performance improved?

Differential Performance

• Exception path of the crash– Sequence of interaction– External input

• Drill down analysis– More instrumentation along the exception path

Crash Analysis