Know-How in Programming Tasks:From Textual Tutorials to

Task-Oriented Knowledge Graph

Helai Bai

A report submitted for the courseCOMP4560 Advanced Computing Project

Supervised by: Dr Zhenchang XingThe Australian National University

June 2019c© Helai Bai 2019

Except where otherwise indicated, this report is my own original work.

Helai Bai1 June 2019

Acknowledgments

First and foremost, I would like to show my sincere gratitude to my advisor Dr.Zhenchang Xing, for his patience, motivation, and immense knowledge. With hisvaluable help and comprehensive guidance in every stage of this project, I couldaccomplish this research and achieve higher professional cognition.

I shall extend my thanks to Mr. Jiamou Sun for his kindness assistant and in-structions along with the whole research project who makes great contribution tomodel design and experiment. I would also express out my appreciation to Ms. RuiChu, who participants in this project and provides me with essential guidance. Herinstructions bring me with primal knowledge of this project.

iii

Abstract

Accomplishing programming tasks usually involves performing multiple activities inlogical order. Activities to solve programming tasks possess different relationships,such as subactivity-of, precede-follow and attributes, such as location, condition, API,code. Referring task-solving activities, relationships and attributes as know-howknowledge, we represent the know-how knowledge as an activity-centric knowledgegraph.

Programming task know-how knowledge is commonly documented in semi-structured textual tutorials. A formative study is conducted on the 20 top-viewedhow-to questions with Android-tagged on Stack Overflow that suggests that develop-ers are facing with three information barriers when discovering and understandingthe task-solving knowledge in textual tutorials, which are incoherent modeling oftask intent, tutorial information overload and unstructured task activity description.

Knowledge graph has been shown to be effective in representing relational knowl-edge and supporting knowledge search in a structured way. Unfortunately, existingknowledge graphs extract only know-what information (e.g., APIs, API caveats andAPI dependencies) from software documentation. In this paper, we devise open in-formation extraction (OpenIE) techniques for extracting candidates of task activities,activity attributes and activity relationships from programming task tutorials.

We apply our approach to Android Developer Guide as a proof-of-concept andthe constructed knowledge graph TaskKG enables activity-centric knowledge search,which is composed with activities in hierarchical taxonomy, three types of activitiesrelationships and five types of activity attributes. A comprehensive evaluation ofTaskKG shows high accuracy of our OpenIE techniques. A user study shows thatthis TaskKG is promising in helping developers finding correct answers of how-toprogramming problems.

v

vi

Contents

Acknowledgments iii

Abstract v

1 Introduction 11.1 Problem Statement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.2 Motivations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.3 Project Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21.4 Contribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31.5 Report Outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

2 Background and Related Work 52.1 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52.2 Related work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

2.2.1 API Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . 52.2.2 Sentence-level Information Retrieval . . . . . . . . . . . . . . . . . 62.2.3 Knowledge Graph . . . . . . . . . . . . . . . . . . . . . . . . . . . 62.2.4 Information Searching on Knowledge Graph . . . . . . . . . . . . 6

2.3 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

3 Formative Study 93.1 Methodology: . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93.2 Result and Analysis: . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

3.2.1 Incoherent modeling of task intent . . . . . . . . . . . . . . . . . . 113.2.2 Tutorial information overload . . . . . . . . . . . . . . . . . . . . . 113.2.3 Unstructured task activity description . . . . . . . . . . . . . . . . 12

3.3 Summery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

4 Design and Implementation 134.1 Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

4.1.1 Activity Frame . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134.1.2 Task-oriented Knowledge Graph . . . . . . . . . . . . . . . . . . . 154.1.3 Activity-centric Knowledge Searching . . . . . . . . . . . . . . . . 154.1.4 Proof-of-concept Design . . . . . . . . . . . . . . . . . . . . . . . . 15

4.2 Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164.2.1 Explicit Notion of Task Know-How Knowledge . . . . . . . . . . 164.2.2 Constructing TaskKG by OpenIE Methods . . . . . . . . . . . . . 17

vii

viii Contents

4.2.3 TaskKG Empowered Activity-Centric Knowledge Search . . . . . 204.3 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

5 Experimental Methodology 235.1 Accuracy of OpenIE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

5.1.1 Activity Extraction . . . . . . . . . . . . . . . . . . . . . . . . . . . 235.1.2 Attribute Extraction . . . . . . . . . . . . . . . . . . . . . . . . . . 245.1.3 Relationship Extraction . . . . . . . . . . . . . . . . . . . . . . . . 24

5.2 Usefulness Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 245.2.1 User Study Design . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

6 Results 276.1 Quality of The Constructed TaskKG . . . . . . . . . . . . . . . . . . . . . 29

6.1.1 Activity Extraction . . . . . . . . . . . . . . . . . . . . . . . . . . . 296.1.2 Attribute Extraction . . . . . . . . . . . . . . . . . . . . . . . . . . 306.1.3 Relationship Extraction . . . . . . . . . . . . . . . . . . . . . . . . 31

6.2 Usefulness Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 316.2.1 Statistic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 316.2.2 Findings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

7 Conclusion 337.1 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

Bibliography 35

Appendices 39

Appendix 1: Project Description 411 Project Title . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 412 Project Description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 413 Learning Objects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

Appendix 2: Study Contract 43

Appendix 3 Description of code 471 Data Processing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 472 Data Preperation for Training . . . . . . . . . . . . . . . . . . . . . . . . . 483 Attention-Based Bi-LSTM . . . . . . . . . . . . . . . . . . . . . . . . . . . 484 Embedding and Matching GUI . . . . . . . . . . . . . . . . . . . . . . . . 485 Acknowledgement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48

Appendix 4 README 491 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 492 Requirement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 493 Code Details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49

3.1 Data Processing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50

Contents ix

3.2 Data Preperation for Training . . . . . . . . . . . . . . . . . . . . . 503.3 Attention-Based Bi-LSTM . . . . . . . . . . . . . . . . . . . . . . . 503.4 Embedding and Matching GUI . . . . . . . . . . . . . . . . . . . . 51

4 Acknowledgement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51

x Contents

List of Figures

4.1 Examples of Semi-structured Textual Tutorials . . . . . . . . . . . . . . . 144.2 Approach Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

6.1 Question-Difficulty Ratings and Answer-Confidence Ratings . . . . . . . 28

1 Independent Study Contract 1 . . . . . . . . . . . . . . . . . . . . . . . . 442 Independent Study Contract 2 . . . . . . . . . . . . . . . . . . . . . . . . 45

xi

xii LIST OF FIGURES

List of Tables

3.1 Statistic of 20 top-viewed, Android-tagged, how-to questions on StackOverflow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

4.1 Examples of Activity versus Non-Activity Sentences . . . . . . . . . . . 184.2 Keywords for Location/Condition/Goal Attributes . . . . . . . . . . . . 19

5.1 Eight Search Tasks in User Study . . . . . . . . . . . . . . . . . . . . . . . 25

6.1 Performance Comparison of User Study . . . . . . . . . . . . . . . . . . . 276.2 Performance of Activity Sentence Classification . . . . . . . . . . . . . . 29

xiii

xiv LIST OF TABLES

Chapter 1

Introduction

This chapter gives the introduction of this research project. Section 1.1 states theproblems that this research project addressed. The motivations of conducting thisresearch are discussed in section 1.2. The project scope and contribution of thisresearch are given in section 1.3 and section 1.4 respectively, and section 1.5 providesthe report outline.

1.1 Problem Statement

In this project, we intend to develop some open information extraction (OpenIE)techniques to extract valuable information from semi-structured textual documents.We will apply our techniques on extracting programming task activities from theAndroid Developer Website as a proof-of-concept, and utilize some evaluation tech-niques to examine the quality of the task activity entities that obtained. Mining,cleaning and tagging are executed for improving the database quality while parsingtextual documents into structured data.

We then attempt to explore an approach of representing the knowledge withknowledge graph and a method for entity-centric searching on it. The knowledgegraph should be activity-centric and task-oriented that entities are task activities,attributes are the properties of activities. To achieve higher entity searching perfor-mance, we further investigate a ranking policy, which considers both knowledgegraph representation and semantic structure, for scoring the matching results. Agraphical user interface (GUI) is developed as an auxiliary tool for model validation,which aims to help developers deal with Android programming tasks. For furtherevaluating the performance of our prototype, we design a user study for intuitiveand quantitative assessments.

1.2 Motivations

Seeking instruction tutorials from internet is a common challenge for developerswhen dealing with unconversant programming tasks with developing tools. However,finding those instructions can be difficult as developers may unable to access or wellunderstand the demanded instructions, though the well-organised guidance of those

1

2 Introduction

tasks have been pre-considered and shared freely. A typical scenario can be obtainedon developing Android. Android Developer website is managed by Google andmaintains a official developing document of Android, which covers the instructionsabout executing majority of Android programming tasks. However, developers stillactive in many forums such as Stack Overflow, which are contributed and maintainedby the community, raising many programming issues of Android. Although some ofthe problems not resolved by the official document, there are indeed many questionsthat can be adequately solved simply by reading official documents. In this case, someproductive OpenIE-based searching techniques are significance to those document-solvable questions, which directly provide answers to developers or help them accessthe demanded instructions faster and more accurate.

Based on the observation that most programming tasks are composed with mul-tiple programming activities in a logical order, we can aggregate, centred at activity,the overall characteristics and integrated process of a task by describing the rela-tionship between task activities. Activities are distinguished among their intrinsiccharacteristics and extrinsic attributes. Different activities should have different activesubjects and/or objects or different actions, with different attributes attached to them.According to the observation above, we intend to build a task-oriented knowledgegraph to represent those tasks. With cogent capabilities in describing data [Paulheim,2017], we attempt to construct a knowledge graph that consisted with two types oftuples, entity-relation-entity and entity-attribute-value, by describing the associationbetween entities as relations (natural or artificial). This formation preserves not onlysemantic information just like classical word-based models but also the spatial struc-ture among entities just like graphical models. Therefore, by transforming knowledgefrom semi-structured textual documents into a structured knowledge graph, we canretain more information about the original knowledge in terms of logical structureand spatial structure compared to traditional models. In the meantime, knowledgegraph can deal with queries of information with complex association, and achievinghigher search quality by understanding user’s intent at the semantic level. Hence aactivity-centric searching approach is developed to apply our model in practice, witha special ranking policy generated among both semantic meaning and logical/spatialstructure.

1.3 Project Scope

In this project, OpenIE techniques are developed to extract interesting informationfrom semi-structured textual documents and convert into a structured form. As ourdata sourcing from Android Developer Website, those OpenIE operations involvecrawling and parsing techniques of the HTML files. To further clean and analyse theextracted data, methods that recognition of imperative sentences and segmentation ofsentence components are generated to resolve the plain text into relations, activitiesand attributes. Hence a activity-centric knowledge graph has been constructed torepresent our data. Some improved activity-centric searching policies are developed

§1.4 Contribution 3

on the knowledge graph and a user study is conducted for quantitative assessment.

1.4 Contribution

This research makes contributions in the following aspects:

• We conducted a formative study to Android-tagged how-to questions on StackOverflow for investigating information barriers about discovering and under-standing programming task know-how knowledge.

• We develop some open information extraction methods to convert know-howknowledge from semi-structured and unstructured textual tutorials into a task-oriented knowledge graph, and develop an activity-centric search engine foranswering programming how-to questions based on the knowledge graph.

• We implement a proof-of-concept tool of our approach with Android DeveloperGuide and conduct some experiments for quality assessment of our knowledgegraph and the usefulness of activity-centric knowledge search.

1.5 Report Outline

There are 7 main chapters in this report. Chapter 1 provides the fundamental intro-duction of this project, which introduces the problems that this project addressed andthe motivation of them. Chapter 2 gives information about the background as well assome related works of the problems and techniques that are involved in this project.A formative study of this project is introduced in Chapter 3. Chapter 4 illustratesthe detailed model design and the approaches to implement it. Chapter 5 describesthe experimental methodology. Chapter 6 demonstrates the result of the experiment.Chapter 7 provides the conclusion and future work of this project.

4 Introduction

Chapter 2

Background and Related Work

2.1 Background

Natural language is the most important carrier of human’s knowledge, which is nor-mally represented in semi-structured and unstructured documents. Extracting valu-able information from textual document is a great challenge in the field of knowledgerepresentation and has widely applications in knowledge searching tasks. However,in the booming information era, massive amounts of information are generated andhence leads to the issue of information accessibility. Finding executable instructionsfor achieving programming tasks is a common challenge for developers dealing withprogramming tasks, especially to those unconversant ones.

Although most issues are pre-considered and solutions are shared along Inter-net, developers are still facing difficulties of accessing and understanding properdemanded resources with several barriers: information isolated from main searchingapproaches, information scattered or buried in lengthy documents and incompre-hensible owing to weak logical description and organization. Knowledge graph hasbecome a key technique for search engines in the past decade [Chu et al., 2017],which is adept at entity-centric searching. Formalising knowledge in a structured andmachine-readable way provides convenience to semantic knowledge representationand searching. OpenIE techniques are required to convert knowledge into relationsand arguments that acts as the first step for deeper text understanding tasks [Gamalloet al., 2012].

2.2 Related work

2.2.1 API Documentation

Many aspects of API documentation have been well studied, including quality is-sues [Robillard, 2009; Uddin and Robillard, 2015; Myers and Stylos, 2016; Ko et al.,2014; Robillard and Deline, 2011] traceability recovery [Ye et al., 2016b; Subrama-nian et al., 2014; Robillard and Chhetri, 2015; Dagenais and Robillard, 2012; Rigbyand Robillard, 2013; Chen and Grundy, 2011], augmenting API documentation withcrowd documentation [Chen and Zhang, 2014; Campos et al., 2016], generating API

5

6 Background and Related Work

documentation [McBurney and McMillan, 2014; Collard et al., 2013; Peng et al., 2018],improving the accessibility of API documentation [Li et al., 2018]. A recent workby Li et al. [Li et al., 2018] is the closet work to ours that both propose knowledgegraph based approaches to improve the knowledge accessibility in API documenta-tion. However, Li et al. construct an API caveat knowledge graph with APIs, APIcaveats and API dependencies as entities, entity attributes and entity relationshipsrespectively. By comparison, our knowledge graph is task oriented, consisting oftask activities, activity attributes (e.g., goal, condition, API, code snippet) and activityrelationships (e.g., task hierarchy, temporal dependency). The two knowledge graphsare complementary to some aspects but in different concerns.

2.2.2 Sentence-level Information Retrieval

Techniques [Treude et al., 2014; Xu et al., 2017; Jiang et al., 2017] of retrieving infor-mation on paragraph- or sentence-level have been developed these years. In termsof information type and granularity, TaskNav by Treude et al. [Treude et al., 2014]is the closet technique to ours. Comparing to TaskNav which extracts task phrasesfrom textual documentation based on keywords and sentence patterns, we train asentence classifier to detect activity sentences. Furthermore, TaskNav still treats taskknowledge as natural language information to support task sentence retrieval, butour work represents task knowledge in a knowledge graph with explicit notion oftask activities, activity attributes and activity relationships, and thus we supportactivity-centric search and knowledge presentation.

2.2.3 Knowledge Graph

Knowledge graph enables the transition from the extracted knowledge by tradi-tional information retrieval techniques to knowledge search and question answer-ing [Lukovnikov et al., 2017; Trivedi et al., 2017; Dai et al., 2016]. In this project, weattempt to achieve this transition for programming task knowledge search by con-structing a task-oriented knowledge graph. Open information extraction methods arecommonly used to extract knowledge graph entities and relationships from text, forexample, DBpedia [Lehmann et al., 2015], YAGO [Suchanek et al., 2007] and Wiki-data [Vrandecic and Krötzsch, 2014] from Wikipedia, How-to knowledge graph [Chuet al., 2017] from WikiHow, Software Technique Graph [Zhao et al., 2017] from StackOverflow TagWiki. These existing works inspire the design of our task-orientedinformation extraction methods on textual tutorials.

2.2.4 Information Searching on Knowledge Graph

Our information extraction methods are generated with a combination of machinelearning based and pattern based text analysis methods. Pattern-based methods aresimilar to the methods in [Treude et al., 2014; Zhao et al., 2017]. Deep learning basedactivity sentence classification is inspired by the recent studies of deep neural net-work in natural language processing [Devlin et al., 2018; Kim, 2014; Vaswani et al.,

§2.3 Summary 7

2017]. We have also seen the application of deep learning techniques to software text,for example, cross-language question retrieval [Chen et al., 2016b], recommendinganalogical libraries and APIs [Chen et al., 2016a], classifying self-admitted technicaldebts [Huang et al., 2018a], duplicate bug detection [Ye et al., 2016b]. Many ap-proaches [Xu et al., 2018; Huang et al., 2018b; Parnin et al.] have been proposed toexploit crowd documentation (e.g., Stack Overflow questions and answers) to supple-ment official documentation. Knowledge graph, once constructed, can be embeddedto obtain vector representations of entities and relationships [Bordes et al., 2013, 2014].The embedding of entities and relationships can be exploited for knowledge reasoningand question answering. For example, Han et al. [Han et al., 2018] constructs a soft-ware weakness knowledge graph which supports the reasoning of software weaknessrelationships and consequences based on knowledge graph embedding. The knowl-edge graph embedding is a promising method we want to apply to task-orientedknowledge graph.

2.3 Summary

The background about the research problem and some techniques of the researchproject is introduced in this chapter. Some related studies of API documents, paragraph-or sentence-level information retrieval, knowledge graph, information searching onknowledge graph. In the next chapter, a formative study is conducted to provide aproof about significance of this research with qualitative and quantitative methods.

8 Background and Related Work

Chapter 3

Formative Study

In this chapter, a formative study is conducted to investigate the accessibility andunderstanding to textual document tutorials of programming tasks. Referring task-solving activities, activity attributes and activity relationships as know-how knowl-edge for programming tasks, we find that those programming task know-how knowl-edge is commonly documented in semi-structural textual tutorials. We hypothesizethat the discovery and understanding of complex know-how knowledge on suchsemi-structure textual representation of programming task know-how knowledge isnot effective. To validate our hypothesis, we investigate 20 how-to questions on StackOverflow (see Table 3.1) which are top-viewed and Android-tagged. These questionshave been viewed 13,503,320 times in total and received 10,329 votes (as of 25th March,2019), which implies the significance of representativeness of these questions.

3.1 Methodology:

For exploring the the difficulties of discovering and understanding programming taskknow-how knowledge in textual tutorials, we conduct a formative study of questionsand answers on Stack Overflow. In particular, we examine the questions about An-droid on Stack Overflow, which have titles starting with “How to", as we are interestedin know-how knowledge and how-to questions normally involve executing tasks. Wechoose questions of Android as the scope of our study owing to the comprehensiveand well-maintained official API documentation on the Android Developer websiteand because Android is the most popular mobile platform discussed on Stack Over-flow which provides ample materials to us. The selected questions are examined in adescending order of their view counts under the rationale that top-viewed questionsgenerally represent the problems that developers frequently encounter in their work.

According to the selection criteria above, 20 qualified questions are selected foranalysis. With the collected statistics of view cont and vote count for each question,we estimate the potential impact of these problems to developers. In problem aspect,we firstly draw a distinguishment according to the problem description that if thequestioners simply post the questions or make some effort on seeking instructions butfailed. In addition, we examine whether the question in introduced and covered by theofficial API document in Android Developer website. For another, the answers aspect,

9

10 Formative Study

we analysis whether the top-voted answers of these 20 how-to questions come fromor can be replaced by the content of some instructions in Android Developer Guide,based on the existence of hyperlines to or paraphrased content from API documents.We also use Google to search the Android Developer website with question titles andkeywords in answers as an auxiliary proof about the accessibility of the task. Oncewe identify some official Android tutorials as the knowledge source of an answer, wecompare the answer content with the tutorial content to see if the answer content isjust an excerpt of tutorial content, or paraphrased tutorial content, or tutorial contentenriched with additional explanation. We also identify whether the tutorial excerptis at the subsection level or specific step level, how many subsection(s) or step(s) areinvolved, and how many words in these subsections or steps. Based on our searchexperience and tutorial statistics, we attempt to identify the information barriers indiscovering and understanding task know-how knowledge from textual tutorials.

Table 3.1: Statistic of 20 top-viewed, Android-tagged, how-to questions on Stack Overflow

Questions (How to ...) AtmpDirExcerot orParaphrased

A.E. Link to Tutorial Tutorial Complexity

display Toast in Android Yes DirExcerpt No The Basics StL, 2St, 48AW

set the Text color of TextView in code Yes No Tutorial

Change Android SDK Path tools No Paraphrased No Update with SDK Manager SuL, 1Su, 626AW

4. hide the title bar for an Activity in XML with

existing custom themeYes DirExcerpt No

Hide the Status Bar on

Android 4.0 and LowerSuL, 2Su, 268AW

5. start new activity on button click No Paraphrased No Build an Intent SuL, 4Su, 396AW

6. change fontFamilyof TextView in Android Yes Paraphrased Yes Adding fonts to a TextView StL, 1St, 17AW

7. add an image to the drawable folder Yes Paraphrased Yes Create action bar or tab icon SuL, 1Su, 454AW

8. make a background 20% transparent on Android No Paraphrased Yes Color SuL, 1Su, 281AW

9. change background color in android app Themes No Paraphrased Yes Styles and Themes SuL, 3Su, 136AW

10. add manifest permission to an application Yes DirExcerpt No Connect to the network StL, 1St, 16AW

11. use putExtra() and getExtra() for string data Yes Paraphrased Yes Start another activity SuL, 2Su, 341AW

12. open the Google Play Store directly from my

Android applicationYes No Tutorial

13. change the status bar color in android No Paraphrased Yes Customize the default theme SuL, 2Su, 521AW

14. manage startActivityForResult on Android No Paraphrased Yes Start the Activity SuL, 2Su, 87AW

15. use SharedPreferences in Android to store,

fetch and edit valuesNo DirExcerpt Yes Save key-value data SuL, 3Su, 198AW

16. run a Runnable thread in Android at defined intervals Yes No Tutorial

17. convert a Drawable to a Bitmap Yes Paraphrased No Load Bitmaps Efficiently SuL, 2Su, 180AW

18. pass an object from one activity to another on Android No Paraphrased Yes Start another activity SuL,2Su, 341AW

19. check certificate name and alias in keystore files Yes No Tutorial

20. send an object from one Android Activity to

another using IntentsYes Paraphrased No Parcelables and Bundles SuL, 2Su, 229AW

1 Abbreviations: Atmp-Attempted; A.E.-Additional Explanation;DirExcept-Direct Excerpt; StL-Step Level, xSt-x Steps, SuL-Subsection Level, xSu-x Subsections, xAW-x Average Words

3.2 Result and Analysis:

Our examination statistics are shown in Table 3.1. We find that the askers of twelvequestions attempted to solve the tasks but failed, and then came to Stack Overflowfor help. According to our result, we do not find relevant tutorials on the AndroidDeveloper website for only four questions (Q2, Q12, Q16, and Q19), yet the official

§3.2 Result and Analysis: 11

Android tutorials contain relevant task-solving knowledge about eight (Q1, Q4, Q6-7,Q10-11, Q17 and Q20) of those twelve questions, which indicates the weak accessi-bility of the document to those questions and the necessity of auxiliary approachesfor developers to access those instructions. In this case, we focus on the questionswith relevant tutorials, among which seven questions have top-voted answers that arejust an excerpt (sometimes paraphrased) of relevant tutorial content on the AndroidDeveloper, and nine questions have top-voted answers that contain tutorial excerptswith some additional explanation provided by the answerers. Our analysis of these18 questions reveals three information barriers that make it difficult for developers todiscover and understand task solving knowledge in textual tutorials.

3.2.1 Incoherent modeling of task intent

As seen in Table 3.1, the intent of a how-to question is a verb phrase which arecomprised by coherent parts such as action, object and some attributes, for example“hide" (action), “title bar" (object) and “in XML" (location) in Question#4. Task activitydescription in tutorials shares the same linguistic characteristics (see Fig. 4.1). Model-ing a task question as a bag of words lose the coherence of task intent, and as a resultit may retrieve textually similar but activity wise irrelevant tutorials, especially whenthe task question is general like “change background color" (Question#9) or longerlike “hide title bar for an Activity in XML with existing custom theme" (Question#4).As examples, searching “change background color" on the Android Developer web-site finds the top-1 result “The Color System", which just introduces the Androidcolor system without describing how to change background color. Another examplesearching the long question title of Question#4 on the Android Developer websitefinds “Styles and Themes", which is a mismatching to the question “hide title barin XML". To overcome the incoherent modeling of task intent, we need to devise acoherent activity-centric representation of task questions and activity descriptions,and support activity-centric intent comparison and matching.

3.2.2 Tutorial information overload

The searching result, among the 16 questions that have relevant tutorials, in theAndroid Developer website with the question titles, can find the most relevant tuto-rials at the top-1 rank for 11 questions (e.g., Question#5/6/7/14/15). However, therelevant tutorials usually contain a large volume of information. Considering Ques-tion#6, searching “change fontfamily of TextView" returns the relevant tutorial Fontsin XML which contains 5 top-level subsections that are font-related tasks that eachcontains several steps with code examples and screenshots. However, the solutionto Question#6 involves only one key step with 17 words but are buried in all thisoverwhelming information. Such information overload makes developers difficultto find the most relevant task-solving knowledge in the tutorial. To overcome thetutorial information overload issue, we need activity-centric knowledge search, ratherthan document-level information retrieval.

12 Formative Study

3.2.3 Unstructured task activity description

As the tutorial examples in Fig. 4.1 shows, for tutorials in documents, althoughcoarse-grained composite activities are expressed in structured content (e.g., nestedheadings), fine-grained task activities are described in unstructured text. A para-graph of unstructured text may involve several fine-grained activities, each of whichmay involve some APIs, conditions, goals, and/or locations. Such convoluted, un-structured task activity descriptions may cause misunderstanding or overlooking ofrelevant task-solving knowledge, which in turn results in errors when answeringhow-to questions (e.g., Question#10). In this case, for unscrambling the intent ofprogramming tasks, we need to convert task-solving knowledge from implicit infor-mation in semi-structured or unstructured text into explicit task activities, activityattributes and activity relationships.

3.3 Summery

The collected statistics indicates that Android developers are frequently facing withsimilar programming problems and need for seeking the task-solving knowledge likequestions and answers above. Among these 20 questions, our analysis confirms thatseven questions have top-voted answers that are just an excerpt (sometimes para-phrased) from relevant programming tutorials on (https://developer.android.com/), andnine questions have top-voted answers that provide tutorial excerpts with additionalexplanation. This observation suggests that Android official tutorials do contain rel-evant know-how knowledge for many how-to questions that developers frequentlyface. However, we identify three barriers (incoherent modeling of task intent, tutorialinformation overload, unstructured task activity description) that make it difficultto discover and understand relevant tutorials and relevant knowledge content intutorials.

Chapter 4

Design and Implementation

In section 4.1, the rationale and procedures of the concept design are introduced. Thesubsection 4.1.1 gives the frame design of each activity. The task-oriented knowledgegraph is described in subsection 4.1.2 and the policy of activity-centric knowledgesearching on the TaskKG is designed in subsection 4.1.3. For the detailed implemen-tation methods in section 4.2, some notions are defined at subsection 4.2.1 which areused in subsection 4.2.2 to construct the TaskKG with extracting activities, attributesand relations from textual tutorials by OpenIE techniques. The searching approach isimplemented in the last subsection 4.2.3.

4.1 Design

4.1.1 Activity Frame

Programming tasks can be tool oriented (e.g.,Create app icons with Image AssetStudio in Fig. 4.1(a)), code oriented (e.g., Use fonts in XML layouts in Fig. 4.1(b)), andthe combination of both (e.g., Start Another Activity in Fig. 4.1(c)). Several commoncharacteristics can be observed from these examples. First, tasks can be broken downinto a hierarchical taxonomy of activities from composite activities to atomic actions.For example, “Using fonts in XML layouts" ⊂ “Adding fonts to a TextView" ⊂ “setfontFamily attribute" in Fig. 4.1(b). Second, activities of solving a programming taskusually need to be executed in a temporal order, such as 1) “select Notification Icons"→ 2) “Select an Asset Type" in Fig 4.1(a). Third, task-solving activities may possesswith many attributes. For example, the step “Open the Properties window to set thefont for the TextView" in Fig. 4.1(b) involves the API TextView and has the goal “seta font for TextView".

Referring task-solving activities, activity attributes and activity relationships asknow-how knowledge for programming tasks, these programming task know-howknowledge is documented in semi-structural textual tutorials as examples in Fig. 4.1from Android Developer website. In such textual tutorials, the hierarchy and tempo-ral order of activities are implicit in the nested headings and ordered or unorderedlists.

13

14 Design and Implementation

(a) Create app icons with Image Asset Studio Excerpt

(b) Use fonts in XML layouts Excerpt

(c) Start Another Activity Excerpt

Figure 4.1: Examples of Semi-structured Textual Tutorials

§4.1 Design 15

4.1.2 Task-oriented Knowledge Graph

To improve the discoverability and comprehensibility of programming task know-how knowledge, we attempt to represent the task activities, activity relationships andactivity attributes as first-class objects in a task-oriented knowledge graph, rather thanas implicit information in document structure and textual description. We parse thecontent of structured documents to infer composite activities, activity hierarchy andactivity temporal order. In addition to composite activities, we train a sentence classi-fier to extract activity sentences in textual descriptions as entities of knowledge graph.For extracting action, object and several attributes in activity sentences, we definethe sentence-patterns based on keywords, sentence annotation, and Part-of-Speech(POS) tags. As a result, a task-oriented knowledge graph (TaskKG) is constructed.Each activity in the knowledge graph is represented as a structured activity frameincluding 1) a verb phrase that describing the activity, 2) action and object in theverb phrase, 3) links to parent activity, sub-activities, sibling, preceding and followingactivities, and 4) values of activity attributes.

4.1.3 Activity-centric Knowledge Searching

For achieving activity-centric knowledge searching, we develop an activity-centricknowledge search engine based on the constructed task-oriented knowledge graph.As we attempt to get know-how knowledge from our TaskKG by querying how-to questions, the search engine firstly resolves the query sentences and models thequestion intent as a activie frame (action, object, activity attributes), in stead of a bag-of-word (as what the traditional search engines do). The engine then finds the mostrelevant matching activity entities in the TaskKG (instead of tutorial documents orfragments) to the question activity. Finally, the engine composes task-solving answersnippets based on matching result and display the snippets to users. These snippetspresent programming task know-how knowledge in well-structured activity frames(see Fig. 4.2), as opposed to unstructured textual descriptions.

4.1.4 Proof-of-concept Design

As a proof-of-concept, we apply our approach on Android Developer Guide to con-struct a task-oriented knowledge graph and a GUI for demonstrate the result. Theresulted TaskKG contains entity (activity) nodes of top, intermediate and leaf levels,relationships of hierarchical, precede-follow and descriptive-sibling, and attributes.We use statistical sampling methods [Singh and Mangat, 1996] to examine the F1-scoreand accuracy of the extracted activities, relationships and attributes. We conduct auser study to inspect the correctness and efficiency of our activity-centric knowledgesearch engine of identifying relevant task-solving knowledge.

16 Design and Implementation

KG Construction

Activity Centric Knowledge Search

TextDescription

Structured Content

RelationshipExtraction

ActivityExtraction

HierarchicalRelation

Activity Sentence

AttributeDetection

ActivityFrame Matching

Top-K RelevantActivity Frames

Compose AnswerSnippets

QueryActivity Frame

Task how- to Questione.g.

How to display Toast?

TemporalRelation

DescriptiveSibling

APIs,Goal,Loc.,CodeCondition

TaskKG

GUI

Instantiate a Toast object

Display the Toast notification

The Basics

Action: DisplayObj: the Toast notification

Code: Toast toast = …

Action: InstantiateObj: a Toast object

VerbPhrase

(action+object)

ConstructActivityFrame

Search

Activity-centric answer snippet

Creating a Custom Toast View

Topic related code:

Verb phrase

Action

Object

Goal None

Condition None

Location None

ActivityRelatedCode

use inflatedlayout to find more Viewuse inflatedlayout to find

more View

None

Verb phrase

Action

Object

Goal None

Condition None

Location None

ActivityRelatedCode

create a new Toast

create

a new Toast

None

Verb phrase

ActionObject

Goal None

Condition None

Location None

ActivityRelatedCode

set some properties of the Toastset

Some properties of Toast

None

Toast toast = new Toast(getApplicationContext()); toast.setView(layout);toast.show();

How to display a Toast? search

HierarchicalRelation

TemporalRelationAttribute

Figure 4.2: Approach Overview

4.2 Implementation

In this project, we implement a knowledge graph based approach to overcome thethree information barriers revealed in our formative study. As shown in Fig. 4.2,the programming task know-how knowledge is represented as explicit task activities,activity attributes and activity relationships (Section 4.2.1), and the activity infor-mation is encapsulated structurally in activity frames. We devise open informationextract (OpenIE) methods to construct a task-oriented knowledge graph (TaskKG)from semi-structured textual tutorials (Section 4.2.2). The resulting TaskKG supportsactivity-centric query intent modeling, candidate activity matching and task know-how knowledge presentation (Section 4.2.3).

4.2.1 Explicit Notion of Task Know-How Knowledge

• Task activity: A programming task is viewed as a hierarchy of activities. Anon-leaf activity can be decomposed into a set of sub-activities. A leaf activity isan atomic action.

• Activity attribute: Five types of activity attributes are designed in this designch:1) API that involved in an activity (e.g., TextView in the activity “set the fontfor TextView"); 2) location where an activity occurs (e.g., in XML file, withinAndroid Studio, from the drop-down list); 3) condition that need to satisfy forperforming an activity (e.g., if you use a new resource qualifier); 4) goal that theactivity aims to achieve (e.g., to preview the UI, so that the highlights aroundthem are clearly visible); 5) code snippet(s) demonstrating one or several activ-ities of the task.

• Activity relationships: Three types of activity relationships are distinguished:1) hierarchical relationship between a parent activity and its sub-activities; 2)

§4.2 Implementation 17

temporal relationship between a preceding sibling activity and a following sib-ling activity with clear step-indicators (e.g., the step 1 “select Notification Icons"precedes the step 2 “Select an Asset Type" in Fig. 4.1(a)); 3) descriptive siblingbetween a sibling activity mentioned immediately before another sibling activ-ity (e.g., “click to place your cursor on the View declaration" before “performa Quick Fix by pressing Alt + Enter" in Fig. 4.1(c)) if no explicit step-indicatorsexits.

• Task activity frame: Three compartments are designed to compose an activityframe which is a structured representation of the information to a task activity:1) verb phrase indicates the action to the object; 2) attributes (API, location,condition, goal, and code) gives the description; and 3) relationships (referencesto parent activity, sub-activities, preceding and following activities, and/ordescriptive-before and descriptive-after activities). An activity frame must hasa verb phrase, while attributes and relationships can be empty.

• Task-oriented knowledge graph (TaskKG): A task-oriented knowledge graphrepresents task know-how knowledge in a direct graph G<T,R> where T is a setof activity frames and R is a set of relationships between activity frames.

4.2.2 Constructing TaskKG by OpenIE Methods

Our approach takes semi-structured textual tutorials as inputs, which document taskknow-how knowledge for application domains, such as Android Developer Guide.Based on the structured content of such tutorials and the linguistic characteristics oftask descriptions, we devise natural language processing techniques and heuristics toextract task activities, activity attributes and activity relationships from the tutorials,which are encapsulated in activity frames to form a task-oriented knowledge graph.

Activity Extraction

Task tutorials in Android Developer Guide describe coarse-grained composite activ-ities in a semi-structured way by document tree and nested headings, and discussfine-grained activities in text, as Fig. 4.1 shows. Coarse-grained composite activitiescan be easily extracted from the structured content, for example by analyzing docu-ment tree (such as <ul> of list structures and <p> of non-list structures) and headings(such as <h1> and <h2>) written in markup languages.

Extracting find-grained activities from unstructured text is the key challenge ofactivity extraction. Based on our observation, fine-grained activities in text are usuallyexpressed in imperative sentences, but not all imperative sentences describe taskactivities. For example, “increase app installation success rate" is an imperativesentence but not an activity. More examples are shown in Table 4.1. To furtherimprove extracting accuracy by overcoming this problem, existing methods [Treudeet al., 2014] defines programming action verbs and object nouns to identify activity

18 Design and Implementation

Table 4.1: Examples of Activity versus Non-Activity Sentences

Candidate Sentences TaskNav

Activity1. Click Start a new Android Studio project. Yes

2. Add a method to the MainActivity class. Yes

3. Tap the Overflow button. No

4. Inflate the string array. No

Non-Activity1. Now take a moment ti review the most important files. No

2. Take note of the details in this method in the TextView. No

3. the system displays a stream notification Yes

4. which it might make sense to request a permission Yes

sentences, and non-programming action verbs and object nouns to filter out non-activity sentences. As indicated in Table 4.1, such pre-defined verb and noun lists areoften too rigid to handle the diversity of activity descriptions. For example, “display"is a programming action verb that defined in [Treude et al., 2014], but “the systemdisplays a stream notification" is not an activity sentence. In contrast, “tap" is anon-programming action verb defined in [Treude et al., 2014], but “tap the Overflowbutton" is an activity sentence.

Therefore, instead of rule-based classification, we develop a sentence classifierfor identifying whether a sentence is an activity sentence or not. Our classifier is anattention-based Bidirectional Long-Short-Term Memory (Bi-LSTM) network [Schusterand Paliwal, 1997], which models bidirectional context of a word, which takes theword embedding representation of a sentence as input. BERT [Devlin et al., 2018]is a start-of-the-art pre-trained word embedding technique that we use to generatethe embedding representation of words for training our sentence classifier. It usesa artificial neural network with 128 hidden states to embed the input sentence. Theoutput sentence vector is fed into a Softmax classifier to classify the sentence intogiven classes.

For each activity sentence, we then parse them to extract the verb phrases in thesentence and detect the action and object of each extracted verb phrase, where theaction can be a verb or verb phrase, and the object can be a noun or noun phrase. Inthe parsing procedure, we firstly use Stanford CoreNLP tool [Manning et al., 2014]to obtain the Part-of-Speech (POS) tags of words in an activity sentence. Then, weadopt a phrase chunking technique [Zhao et al., 2017] for action and object extraction.Specifically, we extract verb (or verb phrase) as action and noun (or noun phrase) asobject based on a set of POS patterns designed for parsing software text. Our POSpatterns handle software-specific verb phrases, such as “click Start a new AndroidStudio Project", which will be chunked into “click" as an action and “Start a newAndroid Studio Project" as an object.

§4.2 Implementation 19

Attribute Extraction

For deeper sentence identification and description, we extract a set of activity at-tributes (if any) from each activity sentence. APIs in a sentence are often annotatedwith special tags (e.g., TextView in “set the font for TextView" in Fig. 4.1(b)). We firstextract candidate text fragments in the activity sentence by looking for such specialtags. If an annotated text fragment is a hyperlink to an API reference document,we extract the annotated text fragment as an API attribute involved in the activity.Furthermore, we use orthographic feature based regular expressions [Ye et al., 2016a]to detect APIs without special tag annotation (e.g., sendMessage() in the activity sen-tence “add the sendMessage() method stub" in Fig. 4.1(c)). These regular expressionsare also used to detect APIs in task how-to questions in the searching phase.

Code snippets extracted from markup language files are always annotated withspecial tags and we can extract them in the same way as APIs in text. A code snippetmay relate to single activity or to a set of composite activities. If a code snippetis immediately after an activity sentence (e.g., the code snippet immediately afterthe step 1 “add the sendMessage() method stub" in Fig. 4.1(c)), we attach the codesnippet to this particular activity as activity-related code. Otherwise, we attached thecode snippet to the parent composite activity, as topic-related code, which usuallyillustrates several sub-activities of the parent activity as a topic summary.

While we manually labelling sentences for training our activity sentence classifier,we also summarize the keywords indicating location, condition or goal attributesin the activity sentences. For each attribute type, Table 4.2 shows the summarizedkeywords and typical examples. Given an activity sentence, we look for these key-words in the sentence and extract the relevant phrase as the location, condition orgoal attribute of an activity by regular expression.

Table 4.2: Keywords for Location/Condition/Goal Attributes

Attribute Keywords Example

Condition if1. If your editor shows the XML source, click theDesign tab at the bottom of the window.

Goalto, in order,

so that

1. To access the updater, use the androidcommand-line tool.

Locationwith, at, on,

near, into, inside,

in from, under, within

1. Click-and-hold the anchor on the top side.2. Change to a custom locate from the adb shell.

Relationship Extraction

The hierarchical relationships of activities are usually implying in document tree,nested content headings and nested lists. As illustrated in Fig. 4.1, the activity at

20 Design and Implementation

an higher level of content structure are parent activities to the activities at one levellower, where lower level activities are direct sub-activities of the parent activity. Forexample, in Fig. 4.1(b), we have “Using fonts in XML layouts" ⊂ “Adding fonts to aTextView" ⊂ “set the fontFamily attribute to the font file you want to access".

Direct sub-activities of the same parent activity are sibling activities, such as “addthe sendMessage() method stub" and “return to the activity main.xml file" in Fig.4.1(c). In this work, we assume that authors of tutorial documents would organizeand explain sibling activities in a logical way, for example, “click to place your cursoron the View declaration", “preform a Quick Fix by pressing Alt + Enter" and “selectImport class" in a paragraph in Fig. 4.1(c). We capture this natural logical orderof sibling activities in the relationship between descriptive siblings. In addition, weheuristically infer a specific type of descriptive sibling relationships in this work torepresent a stronger correlation among task activities, i.e., temporal relationships forsibling activities, which is documented with clear step-like indicators (e.g., the step1 “select Notification Icons" precedes the step 2 “Select an Asset Type" in Fig. 4.1(a)).If no such step-like indicators are present (e.g., the dot list in Fig. 4.1(a)) betweensibling activities, we consider only general descriptive sibling relationships betweenthese sibling activities.

4.2.3 TaskKG Empowered Activity-Centric Knowledge Search

An important usage of knowledge graph is to support entity-centric search with thegoal to “find the things, not the strings" [Lukovnikov et al., 2017]. As activities are theentities in our task-oriented knowledge graph, we develop an activity-centric searchengine which returns relevant task (composite activities) in the TaskKG for an inputhow-to query. The query-activity matching and search results presentation are basedon activity frames (a coherent, structured representation of activity information).

Query-Activity Matching

The given task how-to query is in natural language representation. We firstly extractthe action, object and attribute(s) in the query sentence and formulate into a activityframe using the same activity and attribute parsing techniques as described in Section4.2.2. Then the activity frame of query is matched with the activity frames in ourTaskKG by comparing the verb-phrase and attribute compartments of the activityframes respectively. We separately compare the action phrase and the object phrasefor the verb-phrase compartment, and the API element, location phrase, conditionphrase and goal phrase for the attributes compartment. The matching results ofverb-phrase and attributes compartments are combined to rank the candidate activityframes.

For the comparison and scoring on the verb-phrase compartment over a set ofphrase candidates, we represent each phrase as a embedding vector and calculate thecosine similarity over vectors for sorting over candidates. With the start-of-the-art pre-trained BERT word embedding [Devlin et al., 2018] method, we compute embedding

§4.2 Implementation 21

vector of a phrase by average pooling over the embedding vectors of the words inthe phrases. If the similarity of two phrase is above the predefined threshold (0.8in this work), we consider the pair of candidate phrases as a match. Using wordembedding based phrase representation can mitigate the lexical gap between thequery description and the activity description [Ye et al., 2016b; Chen et al.].

Another matching field is on the attributes compartment. Some attribute compart-ments may be mentioned in the query sentence, but will not contain code snippet.However, an activity frame in the TaskKG can have both textual elements and codesnippet. Therefore, we try to match the textual elements mentioned in the query withall attributes of the activity frames in the TaskKG. As APIs may not always be men-tioned in full-qualified names, we perform partial keyword matching between APInames. As long as one API name is partially matched with the other API name, forexample putExtra and putExtra(String, String[]), startActivityForResult and startAc-tivityForResult(pickContactIntent, PICK CONTACT REQUEST), we consider the twoAPI mentions as a match. This partial keyword matching method has been shown tobe effective on matching API mentions in informal software text [Huang et al., 2018b].

After matching the verb-phrase and attributes compartments of the query activityframe and the activity frames in the TaskKG, we rank the candidate activity framesin the TaskKG by the sum of the similarity scores over all matched compartments ofactivity frames. If the two activity frames have the same similarity score, we furtherrank them by the similarity of sentence-embedding vectors between the how-to queryand the activity description from which an activity frame is extracted. The searchengine then compose answer snippets with the top-k activity frames.

Composing Activity-Centric Answer Snippets

As our approach aims to provide task-solving know-how knowledge, our searchengine is task-oriented and activity-centric, which provides not only the matchedactivity but also the composite activities for coherence. Picking out top-k rankedactivity frames in descending order, the engine composes an answer snippet for eachtop-ranked activity as follows. For a top-ranked activity frame, we traverse its parent-of relationship recursively to find the closest ancestor composite activity extractedfrom structured content (e.g., headings). Furthermore, we traverse the temporalrelationships or the descriptive siblings of the top-ranked activity frame recursivelyto obtain its sibling activities. Ancestor and sibling activities give the context tojudge the relevance of the answer snippet to the query and remind the searchers thenecessary sibling activities around the top-ranked activity. We currently consider up-to four sibling activities (if any) surrounding the top-ranked activity frame (as most ofactivities have four or less sibling activities). If two or more top-ranked activities aresibling activities, their answers will be merged together into as one answer snippet.In such cases, we reduce the number of contextual sibling activities and make roomfor multiple top-ranked sibling activities in the same answer snippet.

We design a user interface (UI) to present activity-centric answer snippets for atask how-to query (see Fig. 4.2). For each answer snippet, the ancestor composite

22 Design and Implementation

activity is shown as a title. The top-ranked activity frame and its contextual siblingactivity frames are shown in structured boxes, which show the verb phrase, action,object and attributes of each activity. The top-ranked activity is shown in red fordistinction. Activity-related code of an activity is shown inside the box of the activityframe, while topic-related code snippet of the ancestor activity is shown underneaththe activity frames. Users can click left- or right-navigation button to see morepreceding (or descriptive before) or following (or descriptive after) sibling activitiesto obtain the contextual activities. Our activity-centric answer snippet UI presentstask-solving knowledge in a more structured way to support intuitive understandingof task-solving knowledge.

4.3 Summary

In this chapter, we introduce the model design of this project and the its detailedimplementation. In the design section, three key concepts are described with exam-ples, which are activity frame, task-oriented knowledge graph and activity-centricknowledge searching. The approaches to implement the model are presented in thesection 4.2. Firstly, we denote some explicit notion of our task know-how knowledge.Then, we illustrate the OpenIE techniques of information extraction to construct theTaskKG, which is followed by an activity-centric knowledge searching method onthat TaskKG.

Chapter 5

Experimental Methodology

5.1 Accuracy of OpenIE

First, we want to assess the accuracy of our information extraction methods by build-ing the confidence in the quality of the resulting task-oriented knowledge graph.

5.1.1 Activity Extraction

Activities are extracted from two main sources: structured document content andactivity textual description. Extracting activities from structured document contentinvolves paring of document tree and heading tags. After carefully implementationand verification of this document parsing step, extracting activities from structureddocument content will not produce erroneous activities. So we mainly evaluate theactivity extraction from text.

Activity Sentence Extraction

We use an attention BiLSTM based sentence classifier to extract activity sentences fromactivity textual description. We create a dataset of activity sentences and non-activitysentences by manual labeling and synonym-based up-sampling [Singh et al., 2018].Using this dataset, we conduct 10-fold cross validation to evaluate the precision, recalland F1-score of our sentence classifier. Precision is the percentage of predicted activitysentences being ground-truth activity sentences. Recall is the percentage of ground-truth activity sentences being correctly classified. F1-score is the harmonic averageof the precision and recall. We use three methods commonly used for sentenceclassification in literature as baselines: linear Support Vector Machine (SVM) [Hearstet al., 1998], TextCNN [Kim, 2014] and Transformer [Vaswani et al., 2017].

Activity Frame Extraction

As our activity frame is composed with complex architecture, we manually examinethe accuracy of activity frames extracted from activity sentences. As examiningall of these activities require significant time and manual efforts, we sample theminimal number MIN of activity frames in the constructed knowledge graph to ensure

23

24 Experimental Methodology

the estimated accuracy in the samples has the 0.05 error margin at 95% confidencelevel [Singh and Mangat, 1996; Li et al., 2018]. For each sampled activity frame,we first independently evaluate the accuracy of its verb phrase (binary decision) bychecking the original activity sentence. Based on the final decisions, we compute theaccuracy for the extracted activity frames. As a comparison, we compare our methodwith a rule-based task phrase extraction method in TaskNav [Treude et al., 2014],which provides an online tool (http://task-phrases.herokuapp.com) for experimentingtheir task phrase extraction method. We manually enter the original activity sentencefrom which a sampled activity frame is extracted into the TaskNav online tool andexamine the accuracy of its extracted task phrases.

5.1.2 Attribute Extraction

For fine-grained modeling of activity information, we extract five types of attributes(API, location, condition, goal and code) from activity sentences and descriptions. Wemanually examine the accuracy of the extracted attributes with statistical samplingmethod described in Section 5.1.1. For each type of attribute, we estimate the accuracyof the extracted attributes for the 0.05 error margin of 95% confidence level. The finalevaluation accuracy is generated over all sampled attribute.

5.1.3 Relationship Extraction

Again, we manually examine the extraction result over three types of activity relation-ships: hierarchical relationships from structured content; descriptive order from thesibling activity mention order in text; and temporal relationships with clear step-likeindicators. Descriptive order is determined by sentence order in text and has noextraction errors. We use the sample statistical sampling method described in Section5.1.1 to examine the MIN number of parent-of, subactivity-of, preceding-followingrelationships respectively for the 0.05 error margin of 95% confidence level. And weexamine each sampled relationship independently to evaluate its accuracy againstthe relevant parts of the tutorial documents.

5.2 Usefulness Evaluation

Next, we want to investigate the usefulness of activity-centric knowledge searchthat our TaskKG empowers for recommending relevant task-solving knowledge forprogramming how-to questions, compared with text-based document or paragraphretrieval.

5.2.1 User Study Design

Search tasks: We design our search tasks in the user study based on the 20 how-toquestions investigated in our formative study. As developers frequently encountering

§5.2 Usefulness Evaluation 25

similar problems as these questions, the search tasks can simulate the real-world infor-mation needs of developers. We select the questions for which developers attemptedthe programming tasks and seek instructions from API documents but failed. Suchquestions have more specific information needs than those questions simply requestsome general task-solving knowledge. Furthermore, we analyze the questions todetermine their difficult levels based on the complexity of programming tasks andthe information volume of relevant tutorials and tutorial contents. Finally, we selecteight questions listed in Table 5.1, including three easy, three moderate and two hardquestions, as our search tasks. We analyze each question and its answers to identifythe specific information needs for the search task (explained in italic font for eachquestion). In our formative study, all eight questions are confirmed to have relevanttask-solving knowledge in Android Developer Guide (see Table 3.1). We use rele-vant tutorial excerpts as the ground-truth answers to the search tasks to evaluate thecorrectness of the participants’ answers.

Table 5.1: Eight Search Tasks in User Study

Task Difficulty

1. How to display Toast in Android? Set duration to LENGTH LONG

and set text to “This is my Toast message!"easy

2. How to hide the title bar for an Activity in XML with existing

custom theme? Assume the android version is 4.0 and lower.easy

3. How to use SharedPreferences in Android to store, fetch and edit

values? Write the process of get edit and apply SharedPreferences.easy

4. How to change fontFamily of TextView in Android? Assume you

want to add the fontFamily âĂIJfont family lightâĂİ to TextView.medium

5. How to change background color in android app? Assume the

color name for background is “@color/white" and you have to add

name “white" with value “#FFFFFF" to the resources.

medium

6. How to add manifest permission to an application? Assume you

get error: WARN/System.err(223):java. net.SocketException:Permission

denied (maybe missing INTERNET permission).

medium

7. How to start new activity on button click? Assume you have to

launch a new activity, and you want to send a message with name

“key" and value named value using putExtra.

hard

8. How to manage startActivityForResult on Android? Write how you

use startActivityForResult() to send the new Intent, and how you

justify the returned result is OK within “onActivityResult".

hard

Participants: 10 senior undergraduate students in our school are recruited byus for this user study. Based on the pre-study background survey, six participants

26 Experimental Methodology

have little Android development experience, and the other four have some Androiddevelopment experience through the school’s one-year industry attachment program.However, none of the participants directly know the answers to the eight searchtasks. We randomly allocate the participants with comparable Android developmentexperience into five pairs. For each pair of participants, they are randomly assignedto use either our activity-centric search engine or Google to complete the search tasksas experimental and control groups respectively.

Baseline: We use Google search engine as the baseline tool, which supports notonly the most powerful document retrieval but also paragraph-level results snippets.A result snippet of Google extracts the most relevant paragraph in one of the topsearch results, which may directly answer the searcher’s question without the needfor reading the whole document. We allow the participants to use Google to searchthe whole Internet and read any materials they want to read, except for the originalStack Overflow questions from which the search tasks are derived. This is to simulatethe situation in which no relevant Stack Overflow questions for the search tasks areavailable, so developers have to find task-solving knowledge in official documentationor other resources.

Experiment procedure: We have an experimental group using our activity-centricsearch engine and a control group using Google. From the user perspective, ouractivity-centric search engine works in the same way as other search engines, takinga natural language query and resulting top ranked search results. We give the ex-perimental group a 10 minutes tutorial of the unique concepts in our tool (activityframe, activity attributes, activity relationships, activity/topic-related code snippets)for helping them start with our tool. As these concepts are very natural, the partici-pants have no difficulty in understanding them. The participants are given the eightsearch tasks one at a time and they complete the task independently. The participantscan formulate any query as they wish based on the search task descriptions and thehints from previous search results. Each task is given up to seven minutes so thatwe can control the experiment in about one hour. For each task, the participantsare asked to submit an answer with detailed information (not just the reference torelevant document) for solving the task. They can copy-paste any document contentthat they find relevant in their answers, but providing irrelevant information will losemarks. At the end of each task, participants are asked to rate the task difficulty andanswer confidence in 5-point scale. We use a screen-recording tool to record the taskcompletion process of participants. From the task videos, we determine the task com-pletion time and observe the participants’ search behavior. After the experiments, wemark the correctness of the participants’ answers against our ground-truth answers.

Chapter 6

Results

The approach is applied on Android Developer Guide for proof-of-concept. Af-ter conducted OpenIE techniques that extracting know-how knowledge from semi-structured tutorials into structured knowledge graph representation, we get a task-oriented knowledge graph with activities, activity relations and activity attributes.We have activities that 950 on top level, 7,722 of intermediate non-leaf compositeactivities and 11,519 atomic leaf. The number of three types of relationships, hierar-chical, precede-follow and descriptive-sibling, are 20,191, 2,870 and 4,524 respectively.In addition, 17,012 activity attributes are generated. In order to examine the qualityof our extraction result, we use statistical sampling methods [Singh and Mangat,1996] as the assessment approach. We also conduct a user study that involving 10participants with eight Android how-to questions on Stack Overflow to evaluate theusefulness of our approach.

Table 6.1: Performance Comparison of User Study

Group participants Avg Task Time (seconds) #Correct Answer

ExperimentalP1 166.6 4

P2 147.8 6

P3 144.9 6

P4 141.6 7

P5 158.8 8

Ave±stddev 151.9±9.3

ControlP6 223.4 3

P7 251.3 5

P8 258.8 7

P9 296.0 7

P10 305.6 8

Ave±stddev 267.0±30.2

27

28 Results

0

2

4

6

8

10

12

14

16

18

Very Easy Easy Morderate Hard Very Hard

Control Group Experimental Group

(a) Question Difficulty

0

2

4

6

8

10

12

14

Lowest Highest

(b) Answer Confidence

Figure 6.1: Question-Difficulty Ratings and Answer-Confidence Ratings

§6.1 Quality of The Constructed TaskKG 29

6.1 Quality of The Constructed TaskKG

We apply our knowledge graph construction methods described in Section 4.2.2 toAndroid Developer Guide. The resulting TaskKG consists of 950 top-level tasks,7,722 intermediate non-leaf activities, and 11,519 atomic actions. These activities have20,191 hierarchical relationships, 4,524 descriptive-sibling relationships, and 2,870temporal relationships. We extract various attributes for these activities: 1,681 goals,1,443 conditions, 7,027 locations, and 4,739 code snippets. We also implement a searchtool as described in Section 4.2.3, which takes a task query as input, searches activityframes in the TaskKG and outputs activity-centric answer snippets for the query.

6.1.1 Activity Extraction

As a brief result, at 95% confidence level and 5% error margin, our information extrac-tion methods achieve 0.97 F1-score for activity sentence extraction, 91.2% accuracyfor activity frame extraction from activity sentences, 87.35% average accuracy forattribute extraction, and 89.54% average accuracy for relationship extraction.

Activity Sentence Extraction

To extract activity sentences from textual documents, we train an attention BiLSTMbased sentence classifier. The classifier is trained and evaluated on a dataset, whichis manually labelled by us, with 29,611 activity sentences and 30,990 non-activitysentences.

Table 6.2: Performance of Activity Sentence Classification

Method Precision Recall F1-score

Our Method 0.96 0.98 0.97

Linear SVM 0.60 0.75 0.65

TextCNN 0.97 0.95 0.96

Transformer 0.96 0.91 0.93

As the result shown in Table 6.2, our method achieves 0.96 precision, 0.98 recalland 0.97 F1-score. As neural network based methods like ours, TextCNN and Trans-former, can extract most informative text features from the input sentences for theclassification task, they all statistically significantly outperform linear SVM whichdirectly classifies the input sentences in their raw form. For activity sentence classifi-cation, our method combining attention mechanism and bidirectional LSTM achievesthe performance on par with the start-fo-the-art CNN-based method TextCNN, andachieves slightly better F1-score than attention-only method Transformer.

30 Results

Activity Frame Extraction

Our constructed knowledge graph contains 13,793 activities extracted from activitysentences, which achieves the accuracy of 91.2% accuracy for the error margin 0.05 at95% confidence level on the sampled 374 activity frames, according to our analysis.The errors mostly owing to the incorrect input activity sentences, which lead tomeaningless activity frames that we consider as false even if it meets our rules. For thecorrect input activity sentences, we achieve 99.0% accuracy of extracting verb phraseand 97.3% accuracy on chunking action and object. These nice results indicates theeffectiveness of our verb phrase extraction and chunking rules. Errors in chunkingaction and object typically involve preposition. The season why our phrase chunkingrules involve prepositions is that they are useful to chunk verb phrases like “try toget" in which actions do involve prepositions. However, such chunking rules resultin some erroneous action/object chunking. For example, “access each of the selectfiles" is chunked into “access each of" as action and “the selected files" as object.

As a comparison, TaskNav achieves 42.2% accuracy of extracting task phrases,which is a low accuracy owing to the rules it implements are rigid and considermany activity sentence as non-programming related (see examples in Table 4.1).Once TaskNav correctly recognizes an activity sentence, it can accurately extract verbphrase in the sentence with a high accuracy (98.1%).

6.1.2 Attribute Extraction

Our method achieves 89.8%, 82.2%, 92.8%, 89.5% and 82.6% accuracy for extractingAPI, location, condition, goal and code snippet, respectively. As we can observe,extracting API, condition and goal is relatively more accurate than extracting locationand code snippet. API elements can be accurately extracted if they are enclosed inhyperlinks to API reference documents, but still many mentions of API elements intext are not enclosed in such hyperlinks. We currently use regular expressions oforthographic features (e.g., camel case, bracket) (similar to [Ye et al., 2016a]) to extractsuch unenclosed API mentions, which can make errors. Such errors can be reducedin future by training a machine-learning based API extraction model [Ye et al., 2016a].

Errors in extracting location, condition and goal are mainly result in our currentsimple keyword matching based attribute extraction method. For example, “to getstarted" will be extracted as a goal, and “if necessary" will be extracted as a condition,but such goals and conditions carry little information so we consider them as errors.Location extraction has more errors because the keywords we defined for it is various(much more than other types of attributes) to handle the diversity of location expres-sions. However, the more keywords used, the more likely to obtain some noisy results.For example, the activity “Use the hasSystemFeature (String) method to check forspecific features at runtime" includes keyword “at", and “at runtime" is erroneouslyextracted as a location. In the future, we will use more robust machine-learningbased method like the one we design for activity sentence extraction to improve theaccuracy of attribute extraction.

§6.2 Usefulness Evaluation 31

As code snippets are always enclosed in special tags, extracting code snippetsfrom text has no errors. But our heuristics (see Section 4.2.2) to link an extractedcode snippet to the corresponding activity may resulting in errors. According to ourdesign, we link a code snippet to the activity immediately before the code snippet, butsometimes the most related activity appears after the code snippet. Another degisnis linking the code snippet to the parent composite activity when the code snippetdoes not appear immediately after an activity. However, when the composite activityhas many sub-activities, it can be difficult to determine which specific sub-activitiesare related to this code snippet. Feature location techniques [Burke, 2014; Alhindawiet al., 2013] may be adopted to improve the accuracy of this traceability recoveryproblem.

6.1.3 Relationship Extraction

For relationship extraction about parent-of, subactivity-of, preceding-following, theaccuracy is 91.7%, 88.2% and 88.8%, respectively. The errors are mainly due tothe errors of activity sentence extraction. For example, if an parent activity is notcorrectly extracted, the sub-activities will be mistakenly linked to the ancestor activity.Similarly, if a sub-activity or sibling activity is not correctly extracted, it will result inwrong subactivity-of or preceding-following relationships.

6.2 Usefulness Evaluation

A user study involving 10 participants and eight Android-related how-to questionsis conducted for examining the usefulness of our approach. The results show thatparticipants using our activity-centric knowledge search engine can correctly identifyrelevant task-solving knowledge with correctness on par with, but 1.76 times fasterthan those using Google search engine.

6.2.1 Statistic

The result in Table 6.1 shows the average task completion time and the number ofcorrect answers by each participant in the two groups. We can see that the two groupshave the on-par answer correctness but the experimental group complete the taskfaster with narrower standard deviation than the control group. Fig. 6.1 shows thetask-difficulty and answer-confidence ratings by the participants in the two groups.We can see that the experimental group participants rate task difficulty as very easy oreasy nine times more than the control group participants, and rate answer confidenceas high or highest six times more than the control group participants.

By reviewing the task videos, we find that Google can return very relevant onlinedocuments for user queries, including not only official Android documentation butalso many other relevant resources (e.g., technical blogs for sharing developmentexperience on specific programming tasks). For example, P3 finds the answer toTask#1 in a technical blog “Android Toast Example". Furthermore, Google provides

32 Results

effective results snippets five times for user queries from which the participants candirectly locate the most relevant paragraphs in the documents for finding answers.Our activity-centric search engine achieves the performance on par with the powerfulGoogle search engine for finding relevant task-solving knowledge for programminghow-to questions.

We also look into the correct answer rate for each search task. Both groups haveonly two correct answers for the two hard search tasks (Task#7/8), which indicatesthat neither our search engine nor Google perform well on these two tasks. Theexperimental group has two more correct answers than the control group for thethree moderate search tasks (Task#4/5/6), but they has one more incorrect answerthan the control group for the three easy search tasks (Task#1/2/3). This incorrectanswer is made by P4 on Task#1 who simply copy-paste the activity-related codesnippet from the answer snippet as the answer without necessary modification.

6.2.2 Findings

Although the two groups’ answer correctness rates are on par, finding correct an-swers in unstructured textual descriptions takes longer time than finding them inthe structured representation of activity information in our activity-centric answersnippets. From the task videos, we observe that the control group participants oftenhave to scroll the documents back and forth even though the answer is just in theparagraph they read. Or they often compare several documents to pinpoint and cross-validate the answer information. In contrast, the experimental group participants canview activity information and activity-related code in a more structured way, whichhelps them find the answers faster, leading to lower task-difficulty ratings and higheranswer-confidence ratings. For example, P5 searches both “change fontFamily onTextview" and “change font on TextView" for Task#4. The relevant information forthe two queries is actually contained in the same tutorial Fonts in XML. Our activity-centric answer snippets present two different sets of activities centered around twodifferent matched activity frames for the two queries. This helps P5 quickly deter-mine the difference between the two seemingly similar queries and the relevance ofthe recommended activities to the search task.

Chapter 7

Conclusion

This project presents a knowledge graph based approach that supports developers“finding activities, not string" for their programming how-to questions. The knowl-edge graph represents task-solving knowledge in explicit task activities, activity at-tributes and activity relationships, extracted from textual programming tutorials us-ing natural language processing techniques. It supports coherent model and matchingof how-to questions and task-solving knowledge by well-structured activity framesand address the information overload issue by activity-centric search. Our experi-ments confirm the effectiveness of our knowledge graph construction method and theusefulness of knowledge graph based activity-centric search for solving programminghow-to questions.

7.1 Future Work

To generalize our results and findings, we will apply our approach to more pro-gramming tutorials and release our knowledge graph and search engine for publicevaluation. Furthermore, we will combine our knowledge graph with other com-plementary software-specific knowledge graphs to build a comprehensive softwareengineering knowledge graph and support advanced knowledge embedding andapplications.

33

34 Conclusion

Bibliography

Alhindawi, N.; Dragan, N.; Collard, M. L.; and Maletic, J. I., 2013. Improving fea-ture location by enhancing source code with stereotypes. In 2013 IEEE InternationalConference on Software Maintenance, 300–309. Ieee. (cited on page 31)

Bordes, A.; Chopra, S.; and Weston, J., 2014. Question answering with subgraphembeddings. arXiv preprint arXiv:1406.3676, (2014). (cited on page 7)

Bordes, A.; Usunier, N.; Garcia-Duran, A.; Weston, J.; and Yakhnenko, O., 2013.Translating embeddings for modeling multi-relational data. In Advances in neuralinformation processing systems, 2787–2795. (cited on page 7)

Burke, J. T., 2014. Utilizing feature location techniques for feature addition andfeature enhancement. In Proceedings of the 29th ACM/IEEE international conference onAutomated software engineering, 879–882. ACM. (cited on page 31)

Campos, E. C.; de Souza, L. B.; and Maia, M. d. A., 2016. Searching crowd knowledgeto recommend solutions for api usage tasks. Journal of Software: Evolution and Process,28, 10 (2016), 863–892. (cited on page 5)

Chen, C.; Gao, S.; and Xing, Z., 2016a. Mining analogical libraries in q&a discussions–incorporating relational and categorical knowledge into word embedding. In 2016IEEE 23rd international conference on software analysis, evolution, and reengineering(SANER), vol. 1, 338–348. IEEE. (cited on page 7)

Chen, C.; Xing, Z.; Liu, Y.; and Ong, K. L. X. Mining likely analogical apis acrossthird-party libraries via large-scale unsupervised api semantics embedding. IEEETransactions on Software Engineering, , 1 , 1–1. (cited on pages 7 and 21)

Chen, C. and Zhang, K., 2014. Who asked what: Integrating crowdsourced faqs intoapi documentation. In Companion Proceedings of the 36th International Conference onSoftware Engineering, 456–459. ACM. (cited on page 5)

Chen, G.; Chen, C.; Xing, Z.; and Xu, B., 2016b. Learning a dual-language vectorspace for domain-specific cross-lingual question retrieval. In 2016 31st IEEE/ACM In-ternational Conference on Automated Software Engineering (ASE), 744–755. IEEE. (citedon page 7)

Chen, X. and Grundy, J., 2011. Improving automated documentation to code trace-ability by combining retrieval techniques. In Proceedings of the 2011 26th IEEE/ACMInternational Conference on Automated Software Engineering, 223–232. IEEE ComputerSociety. (cited on page 5)

35

36 Bibliography

Chu, C. X.; Tandon, N.; and Weikum, G., 2017. Distilling task knowledge from how-to communities. In Proceedings of the 26th International Conference on World Wide Web,805–814. International World Wide Web Conferences Steering Committee. (citedon pages 5 and 6)

Collard, M. L.; Decker, M. J.; and Maletic, J. I., 2013. srcml: An infrastructure forthe exploration, analysis, and manipulation of source code: A tool demonstration.In 2013 IEEE International Conference on Software Maintenance, 516–519. IEEE. (citedon page 6)

Dagenais, B. and Robillard, M. P., 2012. Recovering traceability links between anapi and its learning resources. In Proceedings of the 34th International Conference onSoftware Engineering, 47–57. IEEE Press. (cited on page 5)

Dai, Z.; Li, L.; and Xu, W., 2016. Cfo: Conditional focused neural question answeringwith large-scale knowledge bases. In Proceedings of the 54th Annual Meeting of theAssociation for Computational Linguistics (Volume 1: Long Papers), vol. 1, 800–810.(cited on page 6)

Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K., 2018. Bert: Pre-trainingof deep bidirectional transformers for language understanding. arXiv preprintarXiv:1810.04805, (2018). (cited on pages 6, 18, and 20)

Gamallo, P.; Garcia, M.; and Fernández-Lanza, S., 2012. Dependency-based openinformation extraction. In Proceedings of the joint workshop on unsupervised and semi-supervised learning in NLP, 10–18. Association for Computational Linguistics. (citedon page 5)

Han, Z.; Li, X.; Liu, H.; Xing, Z.; and Feng, Z., 2018. Deepweak: Reasoningcommon software weaknesses via knowledge graph embedding. In 2018 IEEE 25thInternational Conference on Software Analysis, Evolution and Reengineering (SANER),456–466. IEEE. (cited on page 7)

Hearst, M. A.; Dumais, S. T.; Osuna, E.; Platt, J.; and Scholkopf, B., 1998. Supportvector machines. IEEE Intelligent Systems and their applications, 13, 4 (1998), 18–28.(cited on page 23)

Huang, Q.; Shihab, E.; Xia, X.; Lo, D.; and Li, S., 2018a. Identifying self-admittedtechnical debt in open source projects using text mining. Empirical Software Engi-neering, 23, 1 (2018), 418–451. (cited on page 7)

Huang, Q.; Xia, X.; Xing, Z.; Lo, D.; and Wang, X., 2018b. Api method recommen-dation without worrying about the task-api knowledge gap. In Proceedings of the33rd ACM/IEEE International Conference on Automated Software Engineering, 293–304.ACM. (cited on pages 7 and 21)

Bibliography 37

Jiang, H.; Zhang, J.; Ren, Z.; and Zhang, T., 2017. An unsupervised approachfor discovering relevant tutorial fragments for apis. In Proceedings of the 39th In-ternational Conference on Software Engineering, 38–48. IEEE Press. (cited on page6)

Kim, Y., 2014. Convolutional neural networks for sentence classification. arXiv preprintarXiv:1408.5882, (2014). (cited on pages 6 and 23)

Ko, D.; Ma, K.; Park, S.; Kim, S.; Kim, D.; and Le Traon, Y., 2014. Api documentquality for resolving deprecated apis. In 2014 21st Asia-Pacific Software EngineeringConference, vol. 2, 27–30. IEEE. (cited on page 5)

Lehmann, J.; Isele, R.; Jakob, M.; Jentzsch, A.; Kontokostas, D.; Mendes, P. N.;Hellmann, S.; Morsey, M.; Van Kleef, P.; Auer, S.; et al., 2015. Dbpedia–alarge-scale, multilingual knowledge base extracted from wikipedia. Semantic Web,6, 2 (2015), 167–195. (cited on page 6)

Li, H.; Li, S.; Sun, J.; Xing, Z.; Peng, X.; Liu, M.; and Zhao, X., 2018. Improvingapi caveats accessibility by mining api caveats knowledge graph. In 2018 IEEEInternational Conference on Software Maintenance and Evolution (ICSME), 183–193.IEEE. (cited on pages 6 and 24)

Lukovnikov, D.; Fischer, A.; Lehmann, J.; and Auer, S., 2017. Neural network-based question answering over knowledge graphs on word and character level. InProceedings of the 26th international conference on World Wide Web, 1211–1220. Inter-national World Wide Web Conferences Steering Committee. (cited on pages 6and 20)

Manning, C.; Surdeanu, M.; Bauer, J.; Finkel, J.; Bethard, S.; and McClosky, D.,2014. The stanford corenlp natural language processing toolkit. In Proceedings of52nd annual meeting of the association for computational linguistics: system demonstra-tions, 55–60. (cited on page 18)

McBurney, P. W. and McMillan, C., 2014. Automatic documentation generationvia source code summarization of method context. In Proceedings of the 22nd In-ternational Conference on Program Comprehension, 279–290. ACM. (cited on page6)

Myers, B. A. and Stylos, J., 2016. Improving api usability. Communications of theACM, 59, 6 (2016), 62–69. (cited on page 5)

Parnin, C.; Treude, C.; Grammel, L.; and Storey, M.-A. Crowd documentation:Exploring the coverage and the dynamics of api discussions on stack overflow.(cited on page 7)

Paulheim, H., 2017. Knowledge graph refinement: A survey of approaches andevaluation methods. Semantic web, 8, 3 (2017), 489–508. (cited on page 2)

38 Bibliography

Peng, X.; Zhao, Y.; Liu, M.; Zhang, F.; Liu, Y.; Wang, X.; and Xing, Z., 2018.Automatic generation of api documentations for open-source projects. In 2018IEEE Third International Workshop on Dynamic Software Documentation (DySDoc3),7–8. IEEE. (cited on page 6)

Rigby, P. C. and Robillard, M. P., 2013. Discovering essential code elements ininformal documentation. In 2013 35th International Conference on Software Engineering(ICSE), 832–841. IEEE. (cited on page 5)

Robillard, M. P., 2009. What makes apis hard to learn? answers from developers.IEEE software, 26, 6 (2009), 27–34. (cited on page 5)

Robillard, M. P. and Chhetri, Y. B., 2015. Recommending reference api docu-mentation. Empirical Software Engineering, 20, 6 (2015), 1558–1586. (cited on page5)

Robillard, M. P. and Deline, R., 2011. A field study of api learning obstacles.Empirical Software Engineering, 16, 6 (2011), 703–732. (cited on page 5)

Schuster, M. and Paliwal, K. K., 1997. Bidirectional recurrent neural networks.IEEE Transactions on Signal Processing, 45, 11 (1997), 2673–2681. (cited on page 18)

Singh, K.; Sen, I.; and Kumaraguru, P., 2018. Language identification and namedentity recognition in hinglish code mixed tweets. In Proceedings of ACL 2018, StudentResearch Workshop, 52–58. (cited on page 23)

Singh, R. and Mangat, N. S., 1996. Elements of Survey Sampling, vol. 15. SpringerScience & Business Media. (cited on pages 15, 24, and 27)

Subramanian, S.; Inozemtseva, L.; and Holmes, R., 2014. Live api documentation.In Proceedings of the 36th International Conference on Software Engineering, 643–652.ACM. (cited on page 5)

Suchanek, F. M.; Kasneci, G.; and Weikum, G., 2007. Yago: a core of semanticknowledge. In Proceedings of the 16th international conference on World Wide Web,697–706. ACM. (cited on page 6)

Treude, C.; Robillard, M. P.; and Dagenais, B., 2014. Extracting development tasksto navigate software documentation. IEEE Transactions on Software Engineering, 41,6 (2014), 565–581. (cited on pages 6, 17, 18, and 24)

Trivedi, P.; Maheshwari, G.; Dubey, M.; and Lehmann, J., 2017. Lc-quad: A corpusfor complex question answering over knowledge graphs. In International SemanticWeb Conference, 210–218. Springer. (cited on page 6)

Uddin, G. and Robillard, M. P., 2015. How api documentation fails. IEEE Software,32, 4 (2015), 68–75. (cited on page 5)

Bibliography 39

Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.;Kaiser, Ł.; and Polosukhin, I., 2017. Attention is all you need. In Advances inneural information processing systems, 5998–6008. (cited on pages 6 and 23)

Vrandecic, D. and Krötzsch, M., 2014. Wikidata: a free collaborative knowledge-base. Communications of the ACM, 57, 10 (2014), 78–85. (cited on page 6)

Xu, B.; Xing, Z.; Xia, X.; and Lo, D., 2017. Answerbot: Automated generation ofanswer summary to developers’ technical questions. In 2017 32nd IEEE/ACM Inter-national Conference on Automated Software Engineering (ASE), 706–716. IEEE. (citedon page 6)

Xu, B.; Xing, Z.; Xia, X.; Lo, D.; and Li, S., 2018. Domain-specific cross-languagerelevant question retrieval. Empirical Software Engineering, 23, 2 (2018), 1084–1122.(cited on page 7)

Ye, D.; Xing, Z.; Foo, C. Y.; Li, J.; and Kapre, N., 2016a. Learning to extract apimentions from informal natural language discussions. In 2016 IEEE InternationalConference on Software Maintenance and Evolution (ICSME), 389–399. IEEE. (cited onpages 19 and 30)

Ye, X.; Shen, H.; Ma, X.; Bunescu, R.; and Liu, C., 2016b. From word embeddings todocument similarities for improved information retrieval in software engineering.In Proceedings of the 38th international conference on software engineering, 404–415.ACM. (cited on pages 5, 7, and 21)

Zhao, X.; Xing, Z.; Kabir, M. A.; Sawada, N.; Li, J.; and Lin, S.-W., 2017. Hdskg:harvesting domain specific knowledge graph from content of webpages. In 2017IEEE 24th International Conference on Software Analysis, Evolution and Reengineering(SANER), 56–67. IEEE. (cited on pages 6 and 18)

40 Bibliography

Appendix

Appendix 1: Project Description

1 Project Title

Api Knowledge Graph Construction

2 Project Description

API documentation provides important knowledge about API functionality and us-age which helps developers to avoid unintended use of an API with the help ofAPI caveats. However, many important API caveats are scattered in long descrip-tions which is less discoverable. We will try to settle the accessibility issue of APIcaveats by constructing an API caveats knowledge graph from multiple sources ofAPI documentation with an NLP approach.

1 Survey the related work in NLP and knowledge graph;

2 Collect and preprocess software engineering text data;

3 Enhance the information in the existing API caveats knowledge graph;

4 Implement an entity-centric search method based on API caveats knowledgegraph;

5 Conduct empirical evaluation of the entity-centric search;

6 Write technical report.

3 Learning Objects

By conducting this project, the student should have a good understanding of knowl-edge graph and NLP concepts. The student should also gain good skills in construct-ing knowledge graph to solve relevant problems. The student should write a projectreport to demonstrate the knowledge gained in the project.

41

42 Appendix 1: Project Description

Appendix

Appendix 2: Study Contract

43

44 Appendix 2: Study Contract

Research School of Computer Science Form updated Jun 2018

INDEPENDENT STUDY CONTRACT PROJECTS

Note: Enrolment is subject to approval by the course convenor

SECTION A (Students and Supervisors)

UniID: __U6341820__________

SURNAME: __Bai_______________ FIRST NAMES: _______Helai__________________

PROJECT SUPERVISOR (may be external): ______Zhenchang Xing_____________________________

FORMAL SUPERVISOR (if different, must be an RSSCS academic): ___________________________________

COURSE CODE, TITLE AND UNITS: ____COMP4560, Advanced Computing Project, 12 Units_______

COMMENCING SEMESTER S1 S2 YEAR: _2018_ Two-semester project (12u courses only):

PROJECT TITLE:

API Knowledge Graph Construction

LEARNING OBJECTIVES:

By conducting this project, the student should have a good understanding of knowledge graph and NLP

concepts. The student should also gain good skills in constructing knowledge graph to solve relevant problems.

The student should write a project report to demonstrate the knowledge gained in the project.

PROJECT DESCRIPTION: API documentation provides important knowledge about API functionality and usage which helps developers to

avoid unintended use of an API with the help of API caveats. However, many important API caveats are

scattered in long descriptions which is less discoverable. We will try to settle the accessibility issue of API

caveats by constructing an API caveats knowledge graph from multiple sources of API documentation with an

NLP approach.

1. Survey the related work in NLP and knowledge graph;

2. Collect and preprocess software engineering text data;

3. Enhance the information in the existing API caveats knowledge graph;

4. Implement an entity-centric search method based on API caveats knowledge graph;

5. Conduct empirical evaluation of the entity-centric search;

6. Write technical report.

✓ ✓

Figure 1: Independent Study Contract 1

45

Research School of Computer Science Form updated Jun 2018

ASSESSMENT (as per the project course’s rules web page, with any differences noted below).

Assessed project components: % of mark Due date Evaluated by:

Report: style: _______________________________ (e.g. research report, software description...,)

(min 45, def 60)

(examiner)

Lizhen Qu

Artefact: kind: ______________________________ (e.g. software, user interface, robot...,)

(max 45, def 30)

(supervisor)

Zhenchang Xing

Presentation :

(10)

(course convenor)

Peter Strazdins

MEETING DATES (IF KNOWN): STUDENT DECLARATION: I agree to fulfil the above defined contract:

………………………………………………….. ……………………….. Signature Date SECTION B (Supervisor): I am willing to supervise and support this project. I have checked the student's academic record and believe this student can complete the project. I nominate the following examiner, and have obtained their consent to review the report (via signature below or attached email) ………………..………………..……………….. ……………………….. Signature Date Examiner: Name: ………………………………………… Signature …………………… (Nominated examiners may be subject to change on request by the supervisor or course convenor) REQUIRED DEPARTMENT RESOURCES: SECTION C (Course convenor approval) ………………………………………………….. ……………………….. Signature Date

26/July/2018

Figure 2: Independent Study Contract 2

46 Appendix 2: Study Contract

Appendix

Appendix 3 Description of code

There are four main part of code in this project.

1 Data Processing

In this part, we crawled data from the [Android Developer Guide](https://developer.android.com/guide).Then, we use the open information retrieval (OpenIE) techniques that introduced inthe report "**Design and Implementation**" part. Files in this folder is in python 2.

There are eight files and two file folders in this part:

• “1_bone_constructor.py": Create the bone of activity relationship architecture.

• “2_step_construct.py": Parse the extracted plain textual documents into stepactivities. We use stanford corenlp to split sentence. Fill the bone with sentencescontaining steps (activities).

• “3_extract_pre_post.py": Add pre- and post- attributes into the framework.

• “4_extract_parent_fix.py": Clean or regenerate the parent- attributes of someincorrect extracted entities in the matching list [’Audio apps’, ’Messaging apps’,’App quality and distribution’, ’Java’, ’Kotlin’].

• “5_extract_code.py": Extract the code attributes that attached on the activity.

• “6_extract_img.py": Extract the image attributes attached on each activity.

• “7_adjust_img_code.py": As some code and images are not linked correctlypreviously, here we adjust the attachment. Then, we get the complete stepentities and link those entities to bone.

• “8_imperative_deal.py": Extract imperative sentences from steps.

• “attribute_extracting": This folder contains the code files to extract attributesof location, time, goal and condition.

• “outcome": This folder contains all temperal and final data files.

47

48 Appendix 3 Description of code

2 Data Preperation for Training

Codes in this file folder are used to generate the dataset for training the Attention-Based Bi-LSTM model.

3 Attention-Based Bi-LSTM

We develop a sentence classifier to classify whether a sentence is an activity sentenceor not. The classifier is an attention-based Bidirectional Long-Short-Term Memory(Bi-LSTM) network, which models bidirectional context of a word. It takes the wordembedding representation of a sentence as input.

Our methed is developed based on the open source project “Implementation of pa-pers for text classification task on DBpedia(https://github.com/TobiasLee/Text-Classification)"on github. Some modifications are applied to fit our research.

Paper: Attention-Based Bidirectional Long Short-Term Memory Networks forRelation Classification(http://www.aclweb.org/anthology/P16-2034)

4 Embedding and Matching GUI

In this part, we constructed a graphical user interface to demonstrate our approach.To run the demo, simply run the file “GUI_simplify.py".Other files in this folder are used for generating auxiliary data files for speeding

up the GUI’s processing.

5 Acknowledgement

The code in this project is the achievement owing to several contributors. Rui Chu(u6013698@anu.edu.au) and Jiamou Sun (Jiamou.Sun@anu.edu.au) provides the orig-inal version of the construction of the Task-Oriented Knowledge Graph. Jiamou Sunand Helai Bai (u6341820@anu.edu.au) makes further improvement to TaskKG andactivity-centric searching policy on that TaskKG as well as the GUI artefact.

This research project is supervised by Dr. Zhenchang Xing (zhenchang.xing@anu.edu.au).

Appendix

Appendix 4 README

Know-How in Programming Tasks: From Textual Tutorials to Task-Oriented Knowl-edge Graph

1 Overview

With some open information extraction methods we convert semi-structured and un-structured know-how knowledge in textual tutorials into a task-oriented knowledgegraph, and develop an activity-centric search engine for answering programminghow-to questions.

A proof-of-concept tool is implemented using Android Developer Guide andexperiments are conducted to evaluate the the quality of the constructed knowledgegraph and the usefulness of activity-centric knowledge search.

2 Requirement

To run the code, your device must support python2.7. Standord CoreNLP must bepre-installed and in processing.

• python 2.7

• Stanford CoreNLP(https://stanfordnlp.github.io/CoreNLP/)

3 Code Details

There are four parts of code (four folders). Then demo is generated from the last folder"Embedding and Matching GUI". The first three parts are the detailed operations ofthe data about some OpenIE techniques and TaskKG construction (Data Processing)and knowledge embedding methods(Attention-Based Bi-LSTM).

49

50 Appendix 4 README

3.1 Data Processing

In this part, we crawled data from the [Android Developer Guide](https://developer.android.com/guide).Then, we use the open information retrieval (OpenIE) techniques that introduced inthe report "**Design and Implementation**" part. Files in this folder is in python 2.

There are eight files and two file folders in this part:

• “1_bone_constructor.py": Create the bone of activity relationship architecture.

• “2_step_construct.py": Parse the extracted plain textual documents into stepactivities. We use stanford corenlp to split sentence. Fill the bone with sentencescontaining steps (activities).

• “3_extract_pre_post.py": Add pre- and post- attributes into the framework.

• “4_extract_parent_fix.py": Clean or regenerate the parent- attributes of someincorrect extracted entities in the matching list [’Audio apps’, ’Messaging apps’,’App quality and distribution’, ’Java’, ’Kotlin’].

• “5_extract_code.py": Extract the code attributes that attached on the activity.

• “6_extract_img.py": Extract the image attributes attached on each activity.

• “7_adjust_img_code.py": As some code and images are not linked correctlypreviously, here we adjust the attachment. Then, we get the complete stepentities and link those entities to bone.

• “8_imperative_deal.py": Extract imperative sentences from steps.

• “attribute_extracting": This folder contains the code files to extract attributesof location, time, goal and condition.

• “outcome": This folder contains all temperal and final data files.

3.2 Data Preperation for Training

Codes in this file folder are used to generate the dataset for training the Attention-Based Bi-LSTM model.

3.3 Attention-Based Bi-LSTM

We develop a sentence classifier to classify whether a sentence is an activity sentenceor not. The classifier is an attention-based Bidirectional Long-Short-Term Memory(Bi-LSTM) network, which models bidirectional context of a word. It takes the wordembedding representation of a sentence as input.

Our methed is developed based on the open source project “Implementation of pa-pers for text classification task on DBpedia(https://github.com/TobiasLee/Text-Classification)"on github. Some modifications are applied to fit our research.

Paper: Attention-Based Bidirectional Long Short-Term Memory Networks forRelation Classification(http://www.aclweb.org/anthology/P16-2034)

§4 Acknowledgement 51

3.4 Embedding and Matching GUI

In this part, we constructed a graphical user interface to demonstrate our approach.To run the demo, simply run the file “GUI_simplify.py".Other files in this folder are used for generating auxiliary data files for speeding

up the GUI’s processing.

4 Acknowledgement

The code in this project is the achievement owing to several contributors. Rui Chu(u6013698@anu.edu.au) and Jiamou Sun (Jiamou.Sun@anu.edu.au) provides the orig-inal version of the construction of the Task-Oriented Knowledge Graph. Jiamou Sunand Helai Bai (u6341820@anu.edu.au) makes further improvement to TaskKG andactivity-centric searching policy on that TaskKG as well as the GUI artefact.

This research project is supervised by Dr. Zhenchang Xing (zhenchang.xing@anu.edu.au).