Android Crash analysis and The Dalvik Garbage collector – Tools and Tips

Android Crash Analysis and The DalvikGarbage Collector - Tips

Oren BarAd

Mobile Security Team Leader

AVG

2

100,000,000

3

Intro

• App on the market

• Crashes will happen

• Need to Fix the Bugs!

• Identify the root issues

4

Who?

• Developing for Android since 2008

• The first employee of DroidSecurity

– First company to release An antivirus to the Android market (2009)

– Later acquired by AVG (2011)

• Mobile Security Research Team Leader @AVG

• B.Sc Information System Engineering from BGU

5

Agenda

• Android app crash Collection & Analysis

• The Android Garbage Collector

• Memory analysis Tools

• Tips and tricks

6

How to collect crash information?

7

What to fix first?

• Easy to fix

• Affects most users

• The ones users complain most about

• Affects the most popular device

8

Root cause analysis

• Stare at data:

–Crash type

– Exception Type

–Device

–Manufacturer

–OS version

– Locale

• And stare some more…

9

Ultimate debugging method!

• What is the answer to this bug?

– 42

• We don’t know the answer

– Brainstorming

– Developing tools

• Learning something new…

10

Root cause analysis – Real world example

• Concurrent.Timeout exception– ~10% of total crashes for AVG antivirus.

– Unhelpful stack traces

– Second place in top 10 crashes (and number 5, and number 9)

java.util.concurrent.TimeoutException: android.os.BinderProxy.finalize() timed out after 10 seconds

at java.lang.Object.wait(Native Method)

at java.lang.Object.wait(Object.java:401)

at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:102)

at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:73)

at java.lang.Daemons$FinalizerDaemon.run(Daemons.java:170)

at java.lang.Thread.run(Thread.java:841)

java.util.concurrent.TimeoutException: com.android.internal.os.BinderInternal$GcWatcher.finalize() timed out after 10 seconds

java.util.concurrent.TimeoutException: android.os.Parcel.finalize() timed out after 10 seconds

java.util.concurrent.TimeoutException: java.io.FileInputStream.finalize() timed out after 10 seconds

11

Device and OS analysis

33%

66%

Device

Galaxy S3

49%

38%

13%

19%

10%

71%

OSCrash Dashboard

4.2.2

4.3Other OSVersions

12

What next?

• After going deep into the Code…

• Get S3 and use

13

Android GC – little bit of history

Gingerbread

14

Complex Object

Tricolor collector at work

roots rootsroots

object object object

object object objectobject

object

object object

15

object


roots rootsroots



object

object object

16

object


roots rootsroots



object

object object

17

object


roots rootsroots



object

object object

18

Complex Object


roots rootsroots



object

object object

19

Complex Object


roots rootsroots



object

object object

20

Garbage collector observed behavior

Cause Effect

GC_FOR_MALLOC Stop The world GC

Use complex Objects GC takes longer

A lot of small objects GC take Longer

Use more memory GC takes longer

USE_LARGE_HEAP All of the above

21

Trying to replicate the problem

• The GC_Test project– Test the GC performance under load

– Android app to test the behavior of the GC on android

– https://github.com/oba2cat3/GCTest

• Useful logcat python script – https://github.com/oba2cat3/logcat2memorygraph

• Available from GitHub

22

Logcat output

02-04 12:40:33.185: D/dalvikvm(2496): GC_EXPLICIT freed <1K, 19% free 14383K/17720K, paused 1ms+1ms, total 17ms

02-04 12:40:33.265: D/dalvikvm(3279): GC_CONCURRENT freed 661K, 27% free 14934K/20336K, paused 5ms+3ms, total 21ms









02-04 12:40:34.250: D/dalvikvm(3403): GC_EXPLICIT freed 172K, 24% free 14956K/19628K, paused 3ms+4ms, total 28ms






02-04 12:40:34.840: D/dalvikvm(2960): GC_EXPLICIT freed 1236K, 10% free 53524K/58860K, paused 4ms+7ms, total 130ms

02-04 12:40:34.915: D/dalvikvm(5637): GC_FOR_ALLOC freed 143K, 25% free 15978K/21040K, paused 26ms, total 26ms







23

Script result - Heap size and GC delays

0

0.2

0.4

0.6

0.8

1

1.2

0

20000

40000

60000

80000

100000

120000

140000

160000

Time in MSheap size in KB

dalvikvm event timestamps

current heap

max heap

24


0

50

100

150

200

250

300

350

400

0

20000

40000

60000

80000

100000

120000

140000

160000



current heap

max heap

ui pause time

total pause

25


0

50

100

150

200

250

300

350

400

0

20000

40000

60000

80000

100000

120000

140000

160000



current heap

max heap

ui pause time

total pause

webview

start

26

Memory Analyzer Tool tricks

27


28


29


30


31


32


33


34

Tips for managing GC

Do How

Avoid GC_FOR_ALLOC Load stuff in Splash

Use Object Pools For large/heavy objects

Avoid Complex Objects Use simpler Objects

Perform Heap analysis Use MAT and DDMS

Reduce memory use …

35

Summary

• Application will crash

–Collect and Analyze

–Understand and Fix

• Fixing the bug is half the fun

36

Back to the problem

• JNI objects are a special case

• Dalvik will finalize and Destroy

• Timeouts will happen... And they Do!

37

Q&A

• Questions?

• Contact: [email protected]

• We are hiring!

• Come talk to us in the AVG booth

38

Links and resources

• http://android-developers.blogspot.co.il/2011/03/memory-analysis-for-android.html

• http://www.brpreiss.com/books/opus5/html/page424.html

• http://java-is-the-new-c.blogspot.co.il/2013/07/tuning-and-benchmarking-java-7s-garbage.html

• http://www.slideshare.net/muhammedshakirmisarwala/java-performance-monitoring-tuning

• http://www.slideshare.net/VikasBalikai/gc-in-android

• http://www.javacodegeeks.com/2012/01/practical-garbage-collection-part-1.html

39

Links and resources

• https://android.googlesource.com/platform/dalvik/+/android-4.3_r2/vm/alloc/MarkSweep.cpp

• https://android.googlesource.com/platform/dalvik/+log/android-4.4.2_r2/vm/alloc/MarkSweep.cpp

• https://android.googlesource.com/platform/dalvik/+/gingerbread/vm/alloc/MarkSweep.c

• https://android.googlesource.com/platform/dalvik/+/froyo-release/vm/alloc/MarkSweep.c

Android Crash analysis and The Dalvik Garbage collector – Tools and Tips

Technology

Transcript of Android Crash analysis and The Dalvik Garbage collector – Tools and Tips