Macro-scale Mobile App Market Analysis usingCustomized Hierarchical Categorization

Xi LiuMichigan State University

liuxi4@cse.msu.edu

Han Hee SongCisco

hanhsong@cisco.com

Mario BaldiCisco & Politecnico di Torino

mariobal@cisco.com

Pang-Ning TanMichigan State University

ptan@cse.msu.edu

Abstract—Thanks to the widespread use of smart devices,recent years have witnessed the proliferation of mobile appsavailable on online stores such as Apple iTunes and Google Play.As the number of new mobile apps continues to grow at arapid pace, automatic classification of the apps has become anincreasingly important problem to facilitate browsing, searching,and recommending them. This paper presents a frameworkthat automatically labels apps with a richer and more detailedcategorization and uses the labeled apps to study the app market.Leveraging a fine-grained, hierarchical ontology as a guide, wedeveloped a framework not only to label the apps with fine-grained categorical information but also to induce a customizedclass hierarchy optimized for mobile app classification. Withthe classification accuracy of 93%, large-scale categorizationconducted with our framework on 168,000 Google Play appsdiscovers novel inter-class relationships among categories ofGoogle Play market.

I. INTRODUCTION

Smartphones, tablets and other mobile devices have trans-formed the way we communicate, manage, and perform ourdaily tasks. A key driver to the adoption of these devices isthe proliferation of mobile apps developed for personal use,business, education, and other purposes. App markets, such asGoogle Play and Apple iTunes, have provided a convenientplatform for developers to market their creations and for usersto search and download new apps. However, with over threemillion existing apps and 600,000 new apps being introducedto the markets every year [1], organizing and managing themhas become a huge challenge.

A key step towards organizing mobile apps is to assignappropriate categories to each of them. By grouping appsthat are closely related to each other, categorization can helpfacilitate browsing, searching, and recommending apps toaccommodate the greatly varying taste of users. While eachapp market provides its own list of app categories, there areseveral issues with the existing market categorizations.

Firstly, existing categorizations have limited granularity. TheGoogle Play market, for example, provides a flat categorizationof 25 class labels, with an exception of the Games category,which has 6 subclasses. The Apple iTunes store employs asimilar, largely flat categorization as well. While the numberof categories has remained mostly unchanged since the intro-duction of app markets, the explosive growth of mobile appshas resulted in an average of about 15,000 apps per category,rendering the task of searching for an appropriate app in acategory to be laborious and time consuming. The granularity

of current mobile app categorization is also too coarse toeffectively distinguish an app from other apps assigned to thesame category. The lack of details in existing categorizationis detrimental to several analysis efforts, whether focusing onusers or app markets.

The second issue with the current categorization is its lackof objectivity. The classification of mobile apps in onlineapp stores is typically done manually, often based on thesubjective judgment, personal viewpoints, and opinions ofthe app developers. Occasionally, the categories provided bythe developers may not agree with the actual use of theapps. Given the large amount of apps on the markets, it isinfeasible to impose strict guidelines or a revision process tothe categorization. As a result, the apps are not guaranteedto be accurately classified and to share common, undividedsemantics.

The last issue has to do with limited expressiveness. Toprevent developers from overly promoting their apps by as-signing them to multiple classes, markets restrict apps to beexclusively in a single category. As pointed out in [2], thehard, exclusive labeling results in a large number of multi-category apps missing from their appropriate categories. Forexample, the Instagram app [3] can be found only in the socialnetworking category but not in the photography category, eventhough it has functionalities related to both categories.

While the app markets strive to provide the most genericcategorization, the current market categorization limits itsutility in catering the needs of different use cases requiringdifferent viewpoints. For example, to precisely target usergroups, app recommendation needs fine-grained informationon functional similarity among apps (e.g., separating air travelapps from bus and rail; currently all in travel category). Onthe other hand, to prioritize traffic and provision the network,corporate network management focuses on separating businessapps from personal-use apps (e.g., separating Cisco Jabberfrom Facebook messenger, both in communicator category).Having a well-designed customized categorization will flexiblyaccommodate different taxonomic relations and in turn, willgreatly improve our ability to learn what an app is about forthe purpose of app market analysis and user interest research.To overcome these limitations, in this paper, we propose aframework that automatically labels apps with a richer andmore flexible categorization and use the labeled apps to studythe app markets.

In order to label apps with a finer-grained ontology thanthe original categories the app market provides, we leverage adetailed categorization from an application domain that closelyresembles that of mobile apps (e.g., Google Ad Preferenceontology [4]). Unlike the flat structure of the original marketcategorization, our proposed framework is hierarchical so thatboth the generality and specificity between classes are encodedin ancestor-successor node relationships, while classes withcomparable level of generality are encoded in sibling rela-tionships in the class hierarchy. Additionally, to accommodateapps with properties spanning over different classes, ourframework provides multi-class labeling so that such apps canbe accurately labeled. Our categorization framework employsa semi-supervised non-negative matrix factorization approachto simultaneously learn (i) the categories of unlabeled appsand (ii) the inter-class relationships. A hierarchical clusteringalgorithm is then applied to the estimated inter-class relation-ship matrix to induce the class hierarchy.

While the finer-grained categorization should be carefullychosen to overcome the limitation of original market cate-gories, it is not guaranteed to perfectly represent mobile apps.Because its taxonomy is not designed for mobile apps, evenwith an extensive sub-categorization, the fine-grained classesmight not cover all possible concepts and aspects of mobileapps. Similarly, some classes might not be independentlyassociate with the apps nor mutually exclusive of each other.In order to narrow the gap between the apps and the pre-builtinput categorization, our framework re-organizes the inter-class structures of the input categorization and outputs anacclimatized class relationship customized to the mobile apps.

We evaluate the classification accuracy of our frameworkon Google Play market. With 1,065 manually labeled groundtruth, our framework with soft (multi-class) classificationachieves the maximum accuracy of 93.0% with 100% cov-erage whereas a hard classification with Logistic regressionhas 81.4% accuracy at the same coverage level. We alsoevaluated the customized inter-class relationship induced byour framework. Given Google Ad preference tree [4] as an in-stance of the fine-grained input categorization, the frameworkmakes an interesting suggestion on building another layer ofhierarchy by grouping subset of input classes. The finding iscorroborated with a business market research on mobile appuses. Finally, we conducted a macro-scale analysis on GooglePlay market with 166,937 unlabeled apps. In our comparativestudy between original Google Play market categorization andthe finer grained categorization from Google Ad preferences,we observe that the finer grained categorization has higherdescriptive power than the original market category.

II. BACKGROUND

To overcome the limitations of existing categorizationsprovided by app markets, we propose a framework to re-label the apps into finer categories. The framework considersthe following sources of information as input: (i) app relatedinformation that can be used to derive the attributes forclassification, (ii) a new set of fine-grained custom categories

obtained from an external source, and (iii) a method to re-classify a subset of the apps into the custom categories. There-labeled apps along with their corresponding features willbe used to create a training set for building our app classifier.The sources of information along with their characteristics willbe described in further details in the following subsections.Even though the discussion here focuses on Android apps, ourmethodology is also applicable to apps from Apple iTunes.

A. Metadata from Google Play Market

We wrote a script program to collect information on 466,082mobile apps from the Google Play Market [5] using its WebAPI. The API enables us to gather information about eachapp, including the app ID, title, description, and originalmarket category assigned by the developers. The app titleand description were used to derive the textual features forclassifying the apps. Among all the mobile apps we collected,about 20% of the apps either have no description or arewritten in a language other than English. After removingsuch apps, we were left with 374,182 apps. Each of the appsretrieved using Google Play API was assigned to one of 25market categories by the app developers (see Table I). Mostof the market categories do not have sub-categories with theexception of Games, which has a two-level categorization.

TABLE IORIGINAL GOOGLE PLAY MARKET CATEGORIZATION.

Book & Reference Games Personalization LifestyleMedia & Video Tools Business ProductivityTravel & Local Education Communication ComicsMusic & Audio Sports Entertainment MedicalHealth & Fitness Finance Shopping PhotographyNews & Magazines Social Weather TransportationLibraries & Demo

B. Fine-Grained Categorization

Previous studies on document categorization have shownthat acclimatizing a pre-built ontology by human expertsis more effective in real-world applications than buildingcategorizations from scratch [6]. In our research, we use thecategories that appear in the Google Ad tree obtainedfrom the Google Ads Preference Manager [4] as our inputcustom categorization. The tree is originally designed to helponline advertisers to setup targeted ad campaigns based onbrowsing activities of Google and its licensed service users.The tree organizes user interests into a hierarchical structureof up to seven levels of depth. Because many (but not all)of today’s mobile apps are device-specific versions of onlineservices, the tree can serve as a good example of finer-grained,customized categorization for the apps.

The scale of the tree, however, poses a challenge in ourstudy, specifically, in the validation of our results. While ouralgorithm has no restriction in the size of input categorization,finding enough amount of apps for ground truth labelingonto a specific category (e.g., Chrysler, Women’s apparel,Gymnastics) was not financially feasible (Section II-C explainsour method for ground truth building). For the purpose ofresearch, therefore, we resorted to use 82 categories (10

Fig. 1. A snippet of a custom input categorization

intermediate and 72 leaf nodes) from the top two levels (for theabove three example categories, we use their parent categoriesof Cars, Shopping, Individual sports, respectively). While thetree has been dramatically pruned from the original one, it stillis twice larger than the original market categories.

Figure 1 shows a snippet of the tree structure. Noticethat some intermediate nodes are semantically equivalent toclasses in the original Google Play market category (e.g.,News). However, their sub-categories further divides the appsinto finer-grained classes (e.g., Weather, Shopping search, andGossip & tabloid). A complete listing of the categories isshown in Table II.

TABLE IIFINE-GRAINED CUSTOM CATEGORIZATION FROM GOOGLE AD TREE.

Level 1 category labels Level 2 category labels Level 1 category labels Level 2 category labels

Arts & Entertainment

Comics & Animation

News

Business NewsMusic & Audio Weather

TV & Video Shopping SearchE-Books Newspapers

Fun & Trivia Sports NewsHumor Technology NewsMovies Gossip & Tabloid News

Performing Arts Health NewsEvents & Listings

Online Communities

Dating & Personals

Beauty & Fitness

Face & Body Care Social NetworksFashion & Style Photo & Video Sharing

Fitness File Sharing & HostingHair Care Feed Aggregation

Spas & Beauty Services Blogging ResourcesBody Art Forum & Chat Providers

Beauty Pageants Online Goodies

Computers & Electronics Enterprise Technology Online JournalsSoftware Utilities Virtual Worlds

Games

Board Games

Reference

Geographic Reference MuseumsCard Games Language Resources

Puzzles Technical ReferenceTile Games Sports scores statisticsWord Game Directories & Listings

Educational Games General ReferenceTable Games Humanities

Roleplaying Games Libraries & MuseumsParty Games

Travel

Air TravelArcade Games Bus & Rail

Internet & Telecom

Email & Messaging Car Rental & TaxiSearch Engines Carpooling

Teleconferencing Cruises & ChartersWeb Portals Hotels & Accommodation

Web Services Specialty Travel

Jobs & EducationEducation (resources) Tourist Destinations

Job listings Travel AgenciesInternships Travel Guides

C. Re-labeling of Apps using Crowdsourcing

To train a classifier for labeling the apps into fine-grained,custom categories, we need example mappings between theapps and their custom categories (referred to as ground truthfrom here on). We conducted a crowdsourcing campaign onAmazon Mechanical Turk [7] to obtain the ground truth labels.For each app, we display its title and the entire text of appdescription. A worker chooses a label in our custom category

Mobile App Indices

Mob

ile A

pp In

dice

s

100 200 300 400 500 600 700 800 900 1000

100

200

300

400

500

600

700

800

900

10000

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1News

Reference

Online  Communi2es

Arts  &  Entertainment

Jobs  &  Educa2on

Games

Beauty  &  Fitness

Computer  &  Electronics

Travel

Fig. 2. Heat map of cosine similarities among categories

that she thinks is the best match to the app. To ensure the labelsare reliable, we present each app to three different workers.

We randomly selected 2,514 apps from the 25 major cat-egories and 6 sub-categories for Games for labeling by thecrowd workers. Each worker must assign an app to one ofthe custom categories described in Section II-B. Of them, wefind 1,065 apps (42%) have all three workers unanimouslyagree on a single label. In addition, there were 688 apps(27%) in which two of the three workers agree on a singlelabel. For the remaining 761 apps (30%), all three workersassign different labels. To obtain the most reliable results, weprimarily use the 1,065 apps with unanimous agreements forour modeling unless stated otherwise. Although there were 82custom categories available, these 1,065 apps were assignedto only 49 categories by the workers.

Next, we verify whether the apps labeled into the samecustom category share semantically common terms. To do this,we represent the textual feature of each app as a TF.IDF vectorand compute their pairwise cosine similarities. Figure 2 showsthe cosine similarities among keywords between every pairof the unanimously labeled apps. We represent the similarityin terms of the intensity of a heat map (lighter color beinglower similarity, darker color being higher similarity). Fromthe fact that the diagonal components being highlighted, weinfer that the apps labeled in the same custom categorytend to share stronger semantic similarity. For example, appsin News::Weather category share several common terms intheir descriptions — temperature and thermometer. There are,however, some apps from the same category that do not sharemuch common terms (i.e., categories with light color). Theseare apps from categories that have broader and more abstractmeanings, e.g., Reference::General Reference. This analysisallows us to not only determine the categories in which ouralgorithm will not perform well, but also identifies categorieswith clearer, more concrete semantics.

Fig. 3. Framework for mobile app categorization.

III. METHODOLOGY

The categories provided by app markets are often inadequateto meet the needs of various applications. This section presentsa framework to create a custom classification with hierarchical,multi-class labeling. As schematically represented in Figure3, our approach consists of the following three main steps:(i) Data collection in which we collect the following threedata: app metadata from app markets, a pre-built fine-grainedontology as custom categorization, and the labels of apps ontothe custom categorization using crowdsourcing, (ii) Featureextraction in which we convert unstructured natural languagetext into a structured feature set, and (iii) Classifier construc-tion, in which we develop a hierarchical learning algorithm.Details of each step are given below.

A. Data collection

As shown in the top part of Figure 3, our categorizationframework has three types of input: app description keywords,pre-built custom categorization, and ground truth labels. Weextracted app keyword features from the app title and appdescription.

The custom categories from Google Ad tree (Section II-B)are used to provide finer-grained categorization than the orig-inal market categories.

Considering the semantics of the input app descriptionkeywords, our classifier modifies the taxonomy of the inputcategories and suggests a customized hierarchy. An equalnumber of apps from every market class are labeled usingcrowdsourcing (see Section II-C).

B. Feature extraction

The objective of the feature extraction step is to turn raw,unstructured natural language texts about apps into structureddata elements and to create a feature vector for each app.Feature extraction is done in the following three sub-steps:

(i) Word tokenization, (ii) Stemming and lemmatization, (iii)Keyword extraction.

Through (i) and (ii), we translate each word in the text into{word term, occurrence frequency} tuples. We omit detaileddescription of the tokenization and lemmatization methods aswe follow standard methods described in [8].

For (iii), for each app ai, we vectorize the term-frequencytuples into a frequency vector (i.e., feature vector) of length1 × d, where d is the number of all terms that appear in theentire corpus (i.e., in all app descriptions). By concatenating allthe row vectors as columns, we build a matrix X of dimensionn×d, where n is total app count and d is length of each featurevector.

Further, to help alleviate the ill-effect of stop words (e.g.,articles and prepositions) and other uninformative words thatappear frequently in the textual description of apps, we convertthe values of the matrix from term frequency into well-known,Term-Frequency Inverse Document Frequency (TF.IDF ) [9].Each entry Xij corresponds to the TF.IDF value of the jthterm tj of ith app a. The TF.IDF value is computed based onthe equation: Xij = tf(ai, tj) log idf(tj , A), where tf(ai, tj)denotes the frequency of term tj in the text description forapp ai and idf(tj , A) = N/(1 + |{a ∈ A : tj ∈ a}|) is ameasure of importance of term tj with respect to the entireapp description corpus A. |{·}| denotes the cardinality of aset.

C. Classifier Construction

Using both the app-keyword matrix and ground truth cate-gorization, we develop a semi-supervised Non-negative MatrixFactorization (NMF) framework to classify the apps. BaselineNMF is a generic learning paradigm that has been successfullyapplied to solve a wide variety of unsupervised and supervisedlearning problems. The idea behind matrix factorization is toseparate out latent categorical information from large inputmatrices by decomposing them into products of their underly-ing latent factors. The enforcement of non-negativity to matrixfactorization is added because in document categorizationproblems, it is natural to represent affinity between a documentand a class as a non-negative value. In Section III-C1, weexplain the standard NMF. Then in Sections III-C2 and III-C3,we explain how we extended the baseline NMF to performsemi-supervision with side-information.

1) Baseline NMF: NMF was originally developed for un-supervised clustering on an input data matrix X ∈ <n×d+ bysolving the following objective function:

minW,L||X −WLT ||2F s.t. W ≥ 0, L ≥ 0 (1)

where || · ||2F denotes Frobenius norm, LT ∈ <k×d+ is a matrixof basis vectors defined in the lower-dimensional manifold(k < d), and W ∈ <n×k+ is low-rank approximation thatencodes cluster membership of the n apps into k clusters.

The non-negativity constraint in NMF ensures that thecluster membership matrix W is easier to interpret (in termsof degree of closeness of an app to a class). By imposing

appropriate constraints on Eq. (1), the NMF approach can beshown to be mathematically equivalent to the optimizationcriterion used by various algorithms including k-means andspectral clustering [10]. However, the unsupervised NMFapproach does not consider the ground truth labels acquiredthrough crowdsourcing. To overcome this problem, we presenta semi-supervised NMF framework using side informationfrom a custom categorization (e.g., Google Ad tree).

2) Side information from custom categorization: To helpguide the app categorization with custom ground truth cate-gories, the NMF framework can be modified to utilize sideinformation about the classes in the input categories. Specifi-cally, the side information can be represented as a k× k classsimilarity matrix P, where k is the total number of availablecategories. Each entry Pij denotes the similarity betweenclasses i and j, which is obtained by checking their siblingrelationship (i.e, whether i and j share a common parent inGoogle Ad Tree). Formally,

Pij =

{p = 1

# of levels if i and j are siblings,0 otherwise.

(2)

Using P as a regularization constraint, our NMF frameworkcan derive a new similarity matrix that best approximates theinter-class relationships found from the input data.

3) NMF with semi-supervision: The semi-supervised NMFformulation developed in this study takes into account variousfactors, such as the class labels associated with sampledtraining set apps, feature vectors of unlabeled apps, as wellas the side information encoding affinity relationship betweeninput categories. All of this information are integrated intoa unified learning framework that automatically generates (i)the predicted labels for previously uncategorized apps, (ii)the important features characterizing different classes, and(iii) a modified inter-class similarity matrix that better fitsto specific characteristics of mobile apps. Specifically, thesemi-supervised NMF framework developed in this paper isdesigned to minimize the following objective function:

minYu,L,B

||Yl −XlW ||2F + ||Yu −XuW ||2F + β||L||2F

+ γ||B − P ||2F (3)s.t. W = LB, B ≥ 0, L ≥ 0, Yu ≥ 0

where the subscripts l and u denote the labeled and unlabeleddata sets; nl and nu denote the number of labeled and unla-beled examples, respectively. Yl ∈ <nl×k

+ and Yu ∈ <nu×k+

are the class indicator matrices, where k is the number ofclasses. Xl ∈ <nl×d

+ and Xu ∈ <nu×d+ are the feature matrices

associated with the labeled and unlabeled data, respectively.The objective function consists of four terms. The first

two terms, ||Yl − XlW ||2F + ||Yu − XuW ||2F measure theerror of the fitted classification model. Note that although alinear model is employed in this study, the formulation canbe easily extended to a non-linear setting using well-knownkernel trick approach [11]. For semi-supervised learning, thesecond term allows us to incorporate unlabeled data into

the NMF formulation. The parameter matrix W ∈ <d×k+ isfurther decomposed into a product of two matrices, L andB. L ∈ <d×k+ identifies the important features of each classand B ∈ <k×k+ is the class similarity matrix estimated fromthe input similarity matrix P . The third and fourth termsin the objective function are regularizer terms to control themodel complexity. The third term β||L||2F ensures that theapproximated feature matrix L is sparse and concise. Thefourth term γ||B − P ||2F penalizes the model if the matrixB deviates significantly from the input similarity matrix Pobtained from our side information.

By simultaneously learning Yu, L, and B, this unifiedframework seeks to simultaneously generate outputs of thethree objectives mentioned earlier: (i) predicted classificationsfor the previously uncategorized app i in max(Yu[i, ·]), (ii) theimportant features characterizing different classes in L, and(iii) a modified inter-class similarity matrix specific to mobileapps, encoded in the matrix B.

We employ an alternating coordinate descent algorithm [10]to solve the objective function. First, the objective function inEquation 3 can be simplified into the following form:

J = Tr[−Y Tl XlLB −BTLTXTl Yl +BTLTXT

l XlLB

−Y Tu XuLB −BTLTXTu Yu +BTLTXT

uXuLB]

+ β Tr[LTL] + γ Tr[BTB −BTP − PTB]

+ Tr[Y Tu Yu] + constant. (4)

By taking the partial derivative of each variable with respectto L, B, and Yu, we obtain the following:

∂J∂L

= −2[XTl Yl +XT

u Yu]BT

+ 2[XTl Xl +XT

uXu]LBBT + 2βL

∂J∂B

= −2LT [XTl Yl +XT

u Yu]

+ 2LT [XTl Xl +XT

uXu]LB + 2γ(B − P )∂J∂Yu

= 2Yu − 2XuLB

Although the overall objective function is non-convex, itis convex with respect to minimizing each of the unknownsYu, L, and B. The standard gradient descent algorithm canbe applied to iteratively update the matrices from their initialvalues as follows:

L← L+ ηL∂J∂L

and B ← B + ηB∂J∂B

,

Note that Yu = XuLB is non-negative if Xu, L, and Bare non-negative. To ensure non-negativity of L and B, wetransform the additive update used in the original gradientdescent to a multiplicative update by choosing appropriatelearning rates, ηB and ηL. The resulting multiplicative updateformula are

Lij ← Lij ×[XT

l YlBT +XT

u YuBT ]ij

[XTl XlLBBT +XT

uXuLBBT + βL]ij(5)

Bij ← Bij ×(LTXT

l Yl + LTXTu Yu + γP )ij

[LT (XTl Xl +XT

uXu)LB + γB]ij. (6)

Algorithm 1 Mobile App Categorization1: Input: Xl, Xu, Yl, P2: Output: Yu and tree structure T3: t = 0;4: Initialize L(0) and B(0)

5: Y(0)u = XuL

(0)B(0)

6: repeat7: t = t+ 18: L

(t)ij ← Lij × [XT

l YlBT+XT

u YuBT ]ij

[XTl XlLBBT+XT

uXuLBBT+βL]ij

9: B(t)ij ← Bij × (LTXT

l Yl+LTXT

u Yu+γP )ij[LT (XT

l Xl+XTuXu)LB+γB]ij

10: Y(t)u = XuL

(t)B(t)

11: until convergence12: T ← Hierarchical Clustering(B(t))13: return Yu and T

A summary of our proposed method is given in Algorithm1. The algorithm takes in app-term matrices of labeled dataXl, unlabeled data Xu, class indicator Yl, and custom classsimilarity P . As long as B and L are initialized to benon-negative matrices, the resulting B and L will be non-negative upon convergence. Once B is found, we apply thegroup average hierarchical clustering algorithm to infer thehierarchical relationship of the classes.

IV. EXPERIMENTAL EVALUATION

This section presents the experimental results of our pro-posed semi-supervised NMF framework. The framework isevaluated based on the following two criteria—(i) accuracyperformance of its app categorization and (ii) effectiveness ofits class hierarchy customization.

We begin by evaluating the accuracy and coverage ofcategorizing apps into finer-grained Google Ad Category Tree(Section IV-A). Here, we evaluate the performance of thesemi-supervised NMF on both hard classification and softmulti-class classification. We then apply hierarchical clusteringto the estimated class similarity matrix to obtain a modifiedclass hierarchy and analyze the effectiveness of the approachby comparing the obtained hierarchy against the Google AdCategory Tree (Section IV-B). Finally, we present interestinginsights into the structure of mobile app markets through anin-depth analysis of the modified hierarchy (Section IV-C).

A. Performance evaluation of app categorization

In order to have a basis for comparison, the LibLinearsoftware [12] implementation of L2-regularized logistic re-gression [13] is run on the same input data. Experimentalresults presented in this section are obtained through 10-foldcross validation on the 1,065 apps with ground truth labels.Each app is characterized by a TF.IDF feature vector of length7,745.

Parameter setting. Through a series of sensitivity analyses,we found that β parameter that controls the sparsity of thefeature-to-class matrix L achieves its highest accuracy when itis between 104 and 105. The large value of β signifies that the

Fig. 4. Trade off between class prediction accuracy and coverage.

L is sufficiently sparse to avoid model overfitting. For γ thatpenalizes deviation of the estimated class similarity B from theinput similarity P, we find the overall accuracy only changesby 5% when it ranges from 10−3 to 105. This suggests thatwe are able to give freedom to new categorization structureby deviating it from the input categorization structure withoutsacrificing accuracy. We set the default values of the β and γparameters as 5× 104 and 0.2, respectively.

Metrics. To measure the accuracy of the frameworkon the set of classified apps that satisfy the minimumthreshold τ , we define Accuracy as the fraction ofcorrectly classified apps among all the classified apps, i.e.,|correctly classified apps ∩ classified apps|/|classified apps|;and define coverage as |classified apps|/|all apps tested|.

Our semi-supervised NMF framework outputs a class mem-bership matrix Yu for the unlabeled apps. We further applynormalization to Yu by normalizing each row vector with itsrow sum. Each entry Yu(i, j) then provides a membershipscore of unlabeled app i in class j between 0 and 1. For hardclassification (i.e., categorizing an app into only one label), weassign the app to the top score class only if its membershipscore exceeds a minimum threshold τ . The threshold τ allowsthe classifier to refrain from making a prediction unless it isabsolutely confident of its prediction. For soft classification,we select top-r classes for app i among the values of Yu(i, ∗)above τ .

Hard classification accuracy. Employing a score threshold τon the normalized class membership matrix Yu, we plot thetrade-off curve between coverage and accuracy in Figure 4and observe that our (hard) semi-supervised NMF algorithmconsistently outperforms the accuracy of logistic regression byabout 2%. For example, with τ = 0, our semi-supervised NMFachieves accuracy of 83.2% with a coverage of 100%. Forthe same coverage, we obtain 81.4% accuracy using logisticregression. With 70% coverage, accuracy of our scheme raisesto 93.8% while accuracy of logistic regression is at 91.0%.

Soft categorization accuracy. In the case of soft categoriza-tion, our framework outputs the top-r class labels for eachapp. An app is said to be correctly classified if its ground truthlabel matches one of the top-r predicted categories. We presentour soft categorization results for r = 2 and r = 3. Figure 4

Fig. 5. AUC comparison for classes with different number of apps.

shows that our NMF with r = 2 exhibits 90% accuracy, whichis a 7% improvement over hard classification. NMF withr = 3 yields a 93% accuracy, achieving 10% improvement. Inaddition, the soft categorization with r = 3 consistently out-performs hard classification with logistic regression by 12%.Inspection on individual classification results discovered thatthe soft classification approach enables several apps to be cor-rectly assigned to multiple categories. For example, a fashionshopping app with ID com.holosfind.showroom was assignedto both news::shopping search and beauty & fitness::fashion& style. As another example, an instant messaging app withsocial networking capabilities with ID jp.naver.line.androidwas classified into both online communities::social networksand internet & telecom::email & messaging.

Area Under Curve (AUC) for each class. Inspired bythe above results on individual class, we further measureclassification performance for each class by employing awell-known metric, Area Under Curve (AUC) of ReceiverOperating Characteristic (ROC) [14].

Using right y-axis of Figure 5, we plot a histogram of 49Google-ad categories that represents the number of apps ineach category (in black). Using left y-axis of Figure 5, weplot two curves that represents AUC of our NMF for eachclass (in red), and AUC of logistic regression (in blue). Forfairness, we use hard classification for both algorithms. Forlarge classes with more than 50 apps, both NMF and logisticregression yields similarly good AUC performance. However,for smaller classes with less than 50 apps, we observe thatsemi-supervised NMF clearly outperforms logistic regression.We speculate that the introduction of side information helpsNMF to perform well for imbalanced categories. Logisticregression, on the other hand, cannot perform well as it cannotbuild reliable class membership model for classes with smallamount of input data.

B. Evaluation of Modified Class Hierarchy

Apart from improved accuracy in app classification, ourframework provides a unique advantage over existing catego-rization algorithms by proposing a modified category structure.In this section, we analyze the macro-scale structure of GooglePlay market through our fine-grained, mobile app optimized

Compu

ters  &  electronics

Jobs  &  edu

ca4o

n::jo

b  lis4n

gs

Arts  &  

Fig. 6. Overview of modified category hierarchy obtained by hierarchicalclustering on B.

categorization. Specifically, Eq. 3 induces an estimate of thepairwise class similarity matrix B from the data. By applyingthe group average hierarchical clustering method on B, weconstruct a modified class hierarchy as shown in Figure 6.The modified hierarchy contains a number of interestingobservations.

Adequacy of Google Ad tree on app categorization. Beforewe discuss structural changes made by our framework, wemention an observation both held in the input and the modifiedhierarchy. As denoted in braces under x-axis in Figure 6,the majority of second level classes (9 out of 10 classes)retains their original sub-categorical information provided byGoogle Ad tree (except for Jobs & Education: Job Listings,which does not share a common parent as Jobs & Education:Education). The fact that the sub-classes under first levelclasses such as Games, News, Reference, and Travel arepreserved shows that, by and large, Google Ad tree has anadequate taxonomy for textual descriptions on Google Playmarket. The two exceptional subclasses (Jobs & Education:Job Listings and Jobs & Education: Education) exhibit a largedistance because their topics are very different.

Associations among first-level classes. Although we considerthe input Google ad tree comprised of only two levels, the in-duced hierarchy exhibits more complex structure. Specifically,groupings of originally first-level classes such as {Online com-munities, Beauty & fitness, and Travel} and {News, Reference,and Arts & entertainment} suggests that Google Ad tree didnot fully represent affinity among its first level nodes.

Figure 6 shows a closer examination on a pair of exampleclasses, News and arts & entertainment. As it can be seen fromtheir class labels, the key terms of the second-level classes canbe summarized as “media.” Upon inspection on common keyterms across classes, we observe that Arts & entertainmentclass share over twice as many terms (5 terms) with News asthe rest of classes (average of 2 terms).

Through our analysis on pair-wise class comparisons asexemplified above, we notice that there are largely three groupsof classes among the previous first-level classes as denoted inred group labels in Figure 6.

Fig. 7. Hierarchy where games and jobs & education::education join.

1) Social networks & photographs group. We find agroup comprised of Online communities, Beauty & fit-ness, and Travel. The classes share common terms suchas “social networks”, “blogs”, “photos”, and “fashion”.

2) Media group. In the middle of the graph, we find an-other group comprised of News, Reference, and Arts &entertainment. The classes share the following popularterms: “news papers”, “books”, and “(TV) programs”.

3) Game & education group. On the right side of thegraph, we find the last group comprised of Games,Education, and Internet & telecom. They share thefollowing terms: “fun”, “play”, “learn”, and “online”.

Interestingly, we corroborate our finding on these three largergroups of Social networks, Media, and Game with a reportfrom business research [15] stating that the three groups are thetop three user activities on smart devices, collectively spanning80% of all activities. While our research on app market shouldnot directly reflect popularity of apps to users (or frequencyof app use), the high-level similarity signifies that there is anon-negligible relationship between the two.Separation among second-level classes. The induced similar-ity can separate second-level classes that were originally on asingle branch. In Figure 7, we observe a sub-class, Education,which is originally under Jobs & education being separatedfrom the rest of its siblings and gets merged with subclassesof another class, Games::*.

To further investigate the association between these classes,we plot Figure 8 that displays a histogram of terms sharedbetween Jobs & education::education and ten level-1 classes.As shown in the figure, we observe that Games categoryshares more than twice more terms (10 terms) than therest of categories (3 terms on average). Upon inspection ofshared terms between Jobs & education::education and level-2subclasses of Games::*, we observe that Games::Puzzles andGames::Educational games share large number of commonwords (8 and 7 terms, respectively) while the rest of thesubclasses share an average of 2 terms with Jobs & educa-tion::Education. From these results, we can confirm that thekeywords associated with Jobs & education::Education sub-class (e.g., training, teaching, lessons) are semantically closerto Games::Puzzles and Games::Educational games than withits previous-siblings, Job listings and Internships.

C. Macro-scale Analysis of Google Play Market

For our macro-scale experiment, we classify 166,937 un-labeled Google Play apps by applying feature weight W we

Fig. 8. Number of keywords shared between Jobs & education::educationclass and all top-level classes.

trained earlier. Since there is no ground truth label availablefor these apps, instead of calculating the accuracy of theframework, we examine the class distribution of the appsgenerated by our framework and compare it against theiroriginal market categorization assigned by the developers.Figure 9 shows a comparison between the 30 Google Playcategories (24 non-game categories and 6 subcategories ofGames) on the x-axis and the 49 Google-Ad categories on they-axis. Each cell in the heat map represents the number of appsthat belong to the Google Play category that were assignedto the corresponding Google Ad category by our framework.There are several interesting insights provided by this analysis.

First, there are quite a substantial number of Google Playcategories that were mapped to a single Google Ad category.Upon further examination, many of the uniquely mappedlabels are consistent with our general expectation. For instance,almost all the shopping apps from Google Play categories aremapped into News::shopping search from Google-ad; almostall the sports apps from Google Play go into News::sportsnews from Googld-ad, etc. This supports our initial assumptionthat the Google Ad preference tree provides good guidance tothe classification of the mobile apps.

Second, we observe some coarse-grained Google Play cat-egories being mapped into multiple fine-grained categoriesin Google Ad tree. For example, apps from Communica-tion category were separated into three sub-categories ofinternet & telecom::email & messaging, internet & tele-com::teleconferencing, and online & communities::social net-works, all located closely in our hierarchy. In the case of appsfrom shopping category, they categorized into two distant cate-gories of news::shopping search and beauty & fitness::fashion& style. Both of these examples demonstrate the advantageof our framework in providing detailed categorization beyondoriginal market categories.

V. RELATED WORK

App classification. Previous work on mobile app classifi-cation categorizes the apps along various dimensions. [16]categorized the apps into a three-dimensional taxonomy withrespect to the app development process. [17] proposed ahierarchical taxonomy of the apps from a business perspectivewhile [18] defines a taxonomy of apps from the perspectiveof user-app interactions. None of these works consider an

Fig. 9. Mapping between input Google Play categories and output Adcategories.

automated approach for classifying mobile apps, thus limitingthe scalability of these approaches.

To overcome this limitation, a number of categorizationmethods employed machine learning techniques includingmaximum entropy classifier [19], bayesian networks and k-nearest neighbor [20]. The approaches, however, were neitherdesigned to incorporate class hierarchy nor do they reorganizeinput categorization such that the modified structure can betterreflect affinity among input data.

NMF for classification. Mobile apps classification can beregarded as a special class of document classification. In recentyears, the basic NMF framework has received wide attentionfrom the text mining community ([21], [22]) thanks to itseffective handling of large-scale corpora through dimension-ality reduction. Unlike the approach proposed in this paper,the NMF formulation used in [21][22] focused on flat (non-hierarchical) categorization of documents. However, manyreal-world classification problems involve a large number ofclasses that are not entirely independent of each other.

These problems can be naturally casted as a hierarchicallearning problem, where the goal is not only to classify thedata points into their appropriate classes, but also to inducea hierarchical relationship among the classes. In [23], a fasthierarchical document clustering algorithm based on NMFis constructed to generate a nested hierarchy of documentclusters. The method is unsupervised and only considers binarysplits at each iteration. Thus, the method is inapplicable to themobile app categorization problem.

In contrast, [2] proposed a novel recursive NMF approachthat leverages supervision on ground truth labels. The pro-posed scheme automatically learns a labeling tree and conductsmodel for testing on large multi-class classification. However,unlike the framework presented in this paper, the recursiveNMF scheme developed in [2] does not accommodate externalclass hierarchies obtained from other sources. As shown in ourexperimental results, the fine-grained categories obtained fromthe Google Ad Tree helps guiding detailed classification evenfor classes with few training data.

VI. CONCLUSION

In this paper we presented an approach to studying mobileapps according to a categorization structure. Starting from theobservation that native classifications offered by app marketshave limitations due to their coarse-grained nature, we built aframework that customizes the deployed classification hierar-chy to best fit the considered apps. Leveraging a fine-grained,hierarchical ontology as a guide, we employ a semi-supervisedNMF not only to label apps with richer categories, but also touncover inter-class relationships optimized to mobile apps.

Our framework achieved a maximum classification accuracyof 93% with 100% coverage for a subset of Google Play appsfor which we had produced ground truth labels. When appliedto a large-scale categorization of 168,000 Google Play apps,our framework discovered novel inter-class relationships.

REFERENCES

[1] “Number of apps available in leading app stores - gartner,”http://www.gartner.com/newsroom/id/2592315.

[2] L. Liu, P. M. Comar, S. Saha, P.-N. Tan, and A. Nucci, “Recursive nmf:Efficient label tree learning for large multi-class problems,” in IEEEInternational Conf on Pattern Recognition, 2012.

[3] “Instagram - online mobile photo-sharing, video-sharing and socialnetworking service,” http://www.instagram.com.

[4] “Google ads preferences manager,” http://google.com/ads/preferences.[5] “Google play,” https://play.google.com/store/apps.[6] L. Tang, J. Zhang, and H. Liu, “Acclimatizing taxonomic semantics for

hierarchical content classification,” in ACM SIGKDD, 2006.[7] “Amazon mechanical turk,” http://https://www.mturk.com.[8] “Stanford core nlp,” http://nlp.stanford.edu/software/corenlp.shtml.[9] G. Salton and M. J. McGill, Introduction to modern information re-

trieval. McGraw-Hill Book Company, New York, 1983.[10] J. Yoo and S. Choi, “Orthogonal nonnegative matrix tri-factorization for

co-clustering: Multiplicative updates on stiefel manifolds,” Informationprocessing & management, vol. 46, no. 5, pp. 559–570, 2010.

[11] S. S. Keerthi, S. K. Shevade, C. Bhattacharyya, and K. R. K. Murthy,“Improvements to platt’s smo algorithm for svm classifier design,”Neural Computation, vol. 13, no. 3, pp. 637–649.

[12] R.-E. Fan, K.-W. Chang, C.-J. Hsieh, X.-R. Wang, and C.-J. Lin.,“LIBLINEAR: A library for large linear classification,” Journal ofMachine Learning Research, vol. 9, pp. 1871–1874, 2008.

[13] M. Sumner, E. Frank, and M. Hall, “Speeding up logistic model treeinduction,” in Proc of PKDD, 2005.

[14] P.-N. Tan, M. Steinbach, and V. Kumar, Introduction to data mining.Addison Wesley, 2006.

[15] “Here’s what people are doing on their smartphones and tablets- business insider,” http://www.businessinsider.com/heres-what-people-are-doing-on-their-smartphones-and-tablets-2013-3.

[16] H. Kemper and E. Wolf, “Iterative process models for mobile applicationsystems: A framework,” in Proceedings of the International Conferenceon Information Systems, ICIS 2002, 2002, p. 37.

[17] C. Seong Leem, H. Sik Suh, and D. Seong Kim, “A classification ofmobile business models and its applications,” Industrial Management &Data Systems, vol. 104, no. 1, pp. 78–87, 2004.

[18] R. Nickerson, U. Varshney, J. Muntermann, and H. Isaac, “Towards ataxonomy of mobile applications,” in Proc of AMCIS, 2007, p. 338.

[19] H. Zhu, H. Cao, E. Chen, H. Xiong, and J. Tian, “Exploiting enrichedcontextual information for mobile app classification,” in Proc of ACMCIKM, 2012.

[20] B. Sanz, I. Santos, C. Laorden, X. Ugarte-Pedrero, and P. G. Bringas,“On the automatic categorisation of android applications,” in 2012 IEEEConsumer Communications and Networking Conference (CCNC), 2012.

[21] W. Xu, X. Liu, and Y. Gong, “Document clustering based on non-negative matrix factorization,” in Proc of ACM SIGIR, 2003.

[22] F. Shahnaz, M. W. Berry, V. P. Pauca, and R. J. Plemmons, “Documentclustering using nonnegative matrix factorization,” Information Process-ing & Management, vol. 42, no. 2, pp. 373–386, 2006.

[23] D. Kuang and H. Park, “Fast rank-2 nonnegative matrix factorizationfor hierarchical document clustering,” in Proc of ACM SIGKDD, 2013.