Test Report Prioritization to Assist Crowdsourced

Testing Zhenyu Chen, Chunrong Fang, Baowen Xu

State Key Laboratory for Novel Software Technology, Nanjing University

James A. JonesDepartment of Informatics, UC Irvine

Yang Feng,

This work is partially supported by the National Basic Research Program of China (973 Program 2014CB340702), the National Natural Science Foundation of China (61170067, 61373013), and the

National Science Foundation under awards CAREER CCF-1350837 and CCF-1116943.

Problem Domain

• Crowdsourced testing is increasingly popularity

• Testing Companies (such as crowdsourcedtesting.com, pay4bugs.com, and itestin.com)

• Software Corporations (such as Baidu, Alibaba, Tencent)

Problem Domain• Users perform testing tasks and write natural-

language test reports to describe behavior

Problem• Crowdsourced test reports are numerous

• Reported failures must be inspected manually

Example Test Report

Environment Input Output Description

Operating System: Windows 7-64-SP1 OS Version No: MS Windows 6.1.7601 System Language: Chinese Screen Resolution: 1920x1080

Select menu -> options in the browser , set “ad block" closed in the Security page, open the link “http://www.qidian.com/ Default.aspx", and floating ads or ads around the edge of the web page are found; select menu -> options in the browser, set “ad block" enhanced in the security page, open the link to check; switch the browser mode, refresh the page to check again.

When the blocking mode is switched, the number of blocked ads is not consistent with the previous one.

Pilot Case Study

Project Information # Test Reports # Failed Test Reports

# Faults

P1 Baidu-Input, Verson: 3.7 http://shurufa.baidu.com/

274 186 27

P2 Baidu-Browser M18 http://liulanqi.baidu.com/

231 47 22

P3 Baidu-Player http://player.baidu.com/

252 62 18

• In 2013, Chen and Liu conducted pilot study [ICSE-NIER 2014]

• Three software projects in collaboration with Baidu

Observations from Conversations with Baidu Developers

• The number of test reports was quickly too great to efficiently inspect

• Great number of false positives

• Many duplicates (for both true and false positives)

• Crowdsourced tests often performed easier/faster tasks

• Natural-language words were often sufficiently similar for duplicated reports

• Faults tended to "cluster" in functionality

Motivation for Automation

Motivation for Seeking Diversity

Motivation for Using Natural-Language

Techniques

Motivation for Seeking Riskier Reports

Test Report Prioritization

Word Segmentation

• ICTCLAS for word segmentation, which is a widely used Chinese NLP platform.

• Test report Input and Description segmented into words • Words tagged for Part-Of-Speech (POS). • Bi-gram model was introduced to count the classes of

words.

Synonym Replacement

• Used Chinese natural-language library to standardize synonyms and normalize words to help account for slight variations in the ways that different people described the same behavior

Build Keyword Dictionary

• Set of words in the test reports form the keyword dictionary (i.e., the corpus of words across test reports).

• Only nouns and verbs are used. • Words are weighted by their frequency.

Build Keyword Vector

• Vector of boolean values • One vector per test report • Indexed by word in the keyword dictionary

Prioritize Test Reports

• Two primary prioritization strategies: • "Div": Seeking diversity among test reports • "Risk": Seeking riskier test reports

• Hybrid prioritization strategy: • "DivRisk": Balancing these two primary strategies

No. Input Description 输⼊入 描述 Ground Truth

R1 Log in renren.com in the compatibility mode, click "Home“ or send friend "Send Gift", then click back button, and click forward button.

There are some inconsistence between the page after clicking forward button and the page before clicking back button.

兼容模式下登录renren.com,点击“个⼈人主⻚页”或者给好友“送礼物”,然后点击后退按钮之后，点击前进按钮。

点击前进按钮之后⻚页⾯面的内容和点击后退按钮之前不⼀一致。 Not a bug

R2 Enter the compatibility mode, log in renren.com, click the link of some friends, then click "Home" button.

Click back button twice, and then go back to friend's page

进⼊入兼容模式，登录进⼊入renren.com,点击其中的某个好友链接,然后点击“个⼈人主⻚页”的按钮,加载完后点击后退按钮。

点击后退⾄至两次，才能通过后退⾄至好友⻚页⾯面。 Bug 1

R3 Open IE, click tools-opitons-security, set Ad Blocking level to "strong", enter http://soudu.org/ in the address bar.

Floating ads and corner ads haven't been blocked successfully. Floating ads and corner ads haven't been blocked successfully.

打开浏览器，打开⼯工具-选项-安全,⼲⼴广告拦截选择为强⼒力，地址栏输⼊入http://soudu.org/

⻚页⾯面悬浮⼲⼴广告和右下⾓角⼲⼴广告没有被成功拦截。 Bug 2

R4 In compatibility mode in the homepage of baidu.com,input "group on" and search, then search "icecream", search "red bull", click back button twice,and then click forward.

Inconsistence between backing and forwarding operations.

兼容模式下在百度⾸首⻚页的输⼊入框中输⼊入“团购”进⾏行搜索再搜索“冰淇淋”再搜索“红⽜牛”连续两次点击“后退”再点击⼀一次“前进”。

前进后退显⽰示结果不正确。 Bug 1

R5 Open IE, in the maximum window, wait till program is loaded, then switch program, quick and continuously switch screen to maximum or full screen.

Sometimes the task bar is lost, especially when open other IE or system is busy. Move the mouse to some task and wait till the task thumbnail appears, the system restores.

打开浏览器，在最⼤大化状况下，等待程序加载完成再进⾏行程序切换，快速连续进⾏行全屏切换。

有时会出现系统底部任务栏不消失的bug，尤其同时打开别的浏览器或系统繁忙时出现可能性更⼤大，将⿏鼠标移⾄至某个任务上等待任务缩略图出现后，恢复正常。

Not a bug

R6 In IE click menu-options, in the secutity page set ad blocking to "closed", open the link "http://www.narutom.com", ads are detected when loading the page. Then click menu-options, in the security page set ad blocking level to "strong", open the link "http://www.narutom.com" and check.

Failed to block ads. 在浏览器点击菜单->选项，在安全⻚页⾯面下先将⼲⼴广告拦截设为关闭；打开链接http://www.narutom.com/，⺴⽹网⻚页加载中发现⾥里⾯面有弹出⼲⼴广告； 再次点击菜单->选项，在安全⻚页⾯面下先将⼲⼴广告拦截设为强⼒力；再次打开链接http://www.narutom.com/查看。

⼲⼴广告拦截失败。 Not a bug

R7 In IE click menu-options, in the secutity page set ad blocking to "closed", open the link "http://www.qidian.com/Default.aspx",many floating ads and side ads appear. Click menu-options, in the security page set ad blocking level to "strong", open the link again and check;

When the mode of ad blocking is change, the number of blocked ads shows inconsistence.

在浏览器点击菜单->选项，在安全⻚页⾯面下先将⼲⼴广告拦截设为关闭；打开链接http://www.qidian.com/Default.aspx，会显⽰示许多漂浮⼲⼴广告和左右边⼲⼴广告； 再次点击菜单->选项，在安全⻚页⾯面下先将⼲⼴广告拦截设为强⼒力；再次打开链接查看；切换浏览器模式，再次刷新链接查看。

当选择的⼲⼴广告拦截模式改变后，拦截的⼲⼴广告数⺫⽬目不⼀一致。 Not a bug

No. Input Description 输⼊入 描述

R1[Compatibility/v, mode/n, under/f, log/v, renren/en, ./m, com/en, ,, click/nw, “/w, personal/n,

homepage/n, ”/w, or/c, give/p, friend/n, “/w, send/v, gift/n, ”/w, ,, then/c, click/nw, back/v, button/n, after/f, ，/w, click/nw, forward/v, button/n, 。/w]

[click/nw, forward/v, button/n, after/f, page/n, of/uj, content/n, and/c, click/nw, back/v, button/n, before/f,

not/d, consistence/a, 。/w]

[兼容/v, 模式/n, 下/f, 登录/v, renren/en, ./m, com/en, ,, 点击/nw, “/w, 个⼈人/n, 主页/n, ”/w, 或者/c, 给/p, 好友/n, “/w, 送/v, 礼物/n, ”/w, ,, 然后/c, 点击/nw, 后退/v, 按钮/n, 之后/f, ，/w, 点击/

nw, 前进/v, 按钮/n, 。/w]

[点击/nw, 前进/v, 按钮/n, 之后/f, 页⾯面/n, 的/uj, 内容/n, 和/c, 点击/nw, 后退/v, 按钮/n, 之前/f, 不/d, ⼀一致/a, 。/w]

R2 [enter/v, compatibility/v, mode/n, ，/w, log/v, in/v, renren/en, ./m, com/en, ,, click/nw, among/r, of/uj, some/r,friend/n, link/nw, ,, then/c, click/nw, “/w, personal/n, homepage/n, ”/w, of/uj, button/n, ,, load/

nw, done/v, after/f, click/nw, back/v, button/n]

[click/nw, back/v, to/p, twice/m, ，/w, only/d, can/v, through/p, back/v,to/p, friend/n, page/n, 。/w]

[进⼊入/v, 兼容/v, 模式/n, ，/w, 登录/v, 进⼊入/v, renren/en, ./m, com/en, ,, 点击/nw, 其中/r, 的/uj, 某个/r, 好友/n, 链接/nw, ,, 然后/c, 点击/nw, “/w, 个⼈人/n, 主页/n, ”/w, 的/uj, 按钮/n, ,, 加载/nw,

完/v, 后/f, 点击/nw, 后退/v, 按钮/n]

[点击/nw, 后退/v, ⾄至/p, 两次/m, ，/w, 才/d, 能/v, 通过/p, 后退/v, ⾄至/p, 好友/n, 页

⾯面/n, 。/w]

R3[open/v, IE/n, ，/w, open/v, tool/n, -, options/n, -, security/an, ,, ads/n, block/v, select/v, for/p, strong/a,

strength/n, ，/w, address/n, frame/n, input/v, http/en, :, /, /, soudu/en, ./m, org/en, /][page/n, floating/v, ads/n, and/c, corner/nw, ads/n,

not/v, by/p, successfully/a, block/v, 。/w][打开/v, 浏览器/n, ，/w, 打开/v, ⼯工具/n, -, 选项/n, -, 安全/an, ,, ⼴广告/n, 拦截/v, 选择/v, 为/p,

强/a, ⼒力/n, ，/w, 地址/n, 栏/n, 输⼊入/v, http/en, :, /, /, soudu/en, ./m, org/en, /]

[页⾯面/n, 悬浮/v, ⼴广告/n, 和/c, 右下⾓角/nw, ⼴广告/n, 没有/v, 被/p, 成功/a, 拦截/v, 。/

w]

R4 [Compatibility/v, mode/n, under/f, in/p, baidu/m, homepage/m, of/uj, input/v, frame/v, in/f, input/v, “/w, groupon/v, ”/w, ，/w, click/nw, search/v, button/n, ，/w, then/c, respectively/d, search/v, “/w,

icecream/n, ”/w, 、/w, “/w, red-bull/nw, ”/w, ，/w, finally/f, continously/a, twice/m, click/nw, “/w, back/v, ”/w, then/d, click/nw, once/m, “/w, forward/v, ”/w, 。/w]

[click/nw, backforward/nw, button/n, after/f, ，/w, search/v, result/n, not/d, correct/a, 。/w]

[兼容/v, 模式/n, 下/f, 在/p, 百度/m, ⾸首页/m, 的/uj, 输⼊入/v, 框/v, 中/f, 输⼊入/v, “/w, 团购/v, ”/w, ，/w, 点击/nw, 搜索/v, 按钮/n, ，/w, 然后/c, 分别/d, 搜索/v, “/w, 冰淇淋/n, ”/w, 、/w, “/w, 红⽜牛/nw, ”/w, ，/w, 最后/f, 连续/a, 两次/m, 点击/nw, “/w, 后退/v, ”/w, 再/d, 点击/nw, ⼀一次/m,

“/w, 前进/v, ”/w, 。/w]

[点击/nw, 前进后退/nw, 按钮/n, 后/f, ，/w, 搜索/v, 结果/n, 不/d, 正确/a, 。/w]

R5[open/v, IE/n, ，/w, in/p, maximize/v, condition/n, down/f, ，/w, wait/v, program/n, load/nw, finish/v, then/d, do/v, program/n, switch/v, ，/w, quick/d, continusly/a, do/v, full-screem/nw, switch/v, 。/w]

[sometimes/d, can/v, appear/v, system/n, bottom/f, task/n, frame/n, not/d, missed/v, of/uj, bug/en, ，/w, especially/d, meanwhile/c, open/v, others/r, IE/n, or/c, system/n, busy/a, time/ng, appear/v, possibility/n, more/d, stronger/a, ，/w, put/d, mouse/n, move/v, to/

p, some/r, task/n, up/f, wait/v, task/n, min/v, thumbnail/n, appear/v, after/f, ，/w, return/v, normal/

a, 。/w]

[打开/v, 浏览器/n, ，/w, 在/p, 最⼤大化/v, 状况/n, 下/f, ，/w, 等待/v, 程序/n, 加载/nw, 完成/v, 再/d, 进⾏行/v, 程序/n, 切换/v, ，/w, 快速/d, 连续/a, 进⾏行/v, 全屏/nw, 切换/v, 。/w]

[有时/d, 会/v, 出现/v, 系统/n, 底部/f, 任务/n, 栏/n, 不/d, 消失/v, 的/uj, bug/

en, ，/w, 尤其/d, 同时/c, 打开/v, 别的/r, 浏览器/n, 或/c, 系统/n, 繁忙/a, 时/ng, 出现/v, 可能性/n, 更/d, ⼤大/a, ，/w, 将/d, ⿏鼠标/n, 移/v, ⾄至/p, 某个/r, 任务/n, 上/f, 等待/v, 任务/n, 缩/v, 略图/n, 出现/v, 后/

f, ，/w, 恢复/v, 正常/a, 。/w]

R6 [in/p, IE/n, click/nw, menu/n, -, >, options/n, ，/w, in/p, security/an, page/n, down/f, firstly/d, put/d, ads/n, block/v, set/v, as/p, close/v, ；/w, open/v, link/nw, http/en, :, /, /, www/en, ./m, narutom/en, ./m, com/en, /, ，/w, page/n, load/nw, in/f, find/v, in/f, has/v, pop/v, out/v, ads/n, ；/w, , again/d, click/nw, menu/n, -, >, options/n, ，/w, in/p, security/an, page/n, down/f, first/d, put/d, ads/n, block/v, set/v, as/p, strong/n, ；/w, again/d, open/v, link/nw, http/en, :, /, /, www/en, ./m, narutom/en, ./m, com/en, /, check/

v, 。/w]

[ads/n,block/v, fail/v, 。/w]

[在/p, 浏览器/n, 点击/nw, 菜单/n, -, >, 选项/n, ，/w, 在/p, 安全/an, 页⾯面/n, 下/f, 先/d, 将/d, ⼴广告/n, 拦截/v, 设/v, 为/p, 关闭/v, ；/w, 打开/v, 链接/nw, http/en, :, /, /, www/en, ./m, narutom/

en, ./m, com/en, /, ，/w, ⽹网页/n, 加载/nw, 中/f, 发现/v, ⾥里⾯面/f, 有/v, 弹/v, 出/v, ⼴广告/n, ；/w, , 再次/d, 点击/nw, 菜单/n, -, >, 选项/n, ，/w, 在/p, 安全/an, 页⾯面/n, 下/f, 先/d, 将/d, ⼴广告/n, 拦截/v, 设/v, 为/p, 强⼒力/n, ；/w, 再次/d, 打开/v, 链接/nw, http/en, :, /, /, www/en, ./m, narutom/en,

./m, com/en, /, 查看/v, 。/w]

[⼴广告/n, 拦截/v, 失败/v, 。/w]

R7 [in/p, IE/n, clcik/nw, menu/n, -, >, options/n, ，/w, in/p, security/an, page/n, down/f, fisrtly/d, put/d, ads/n, block/v, set/v, as/p, close/v, ；/w, open/v, link/nw, http/en, :, /, /, www/en, ./m, qidian/en, ./m,

com/en, /, default/en, ./m, aspx/en, ，/w, can/v, appear/v, many/m, floating/v, ads/n, and/c, left-right/m, side/d, ads/n, ；/w, , again/d, click/nw,menu/n, -, >, options/n, ，/w, in/p, security/an, page/n, down/f,

firstly/d, put/d, ads/n, block/v, set/v, as/p, strong/n, ；/w, again/d, open/v, link/nw, check/v, ；/w, switch/v, IE/n, mode/n, ，/w, click/nw, refresh/v, button/n, check/v, 。/w]

[when/p, select/v, of/uj, ads/n, block/v, mode/n, change/v, after/f, ，/w, block/v, of/uj, ads/n, number/

n, not/d, consistence/a, 。/w]

[在/p, 浏览器/n, 点击/nw, 菜单/n, -, >, 选项/n, ，/w, 在/p, 安全/an, 页⾯面/n, 下/f, 先/d, 将/d, ⼴广告/n, 拦截/v, 设/v, 为/p, 关闭/v, ；/w, 打开/v, 链接/nw, http/en, :, /, /, www/en, ./m, qidian/en, ./m, com/en, /, default/en, ./m, aspx/en, ，/w, 会/v, 显⽰示/v, 许多/m, 漂浮/v, ⼴广告/n, 和/c, 左右/m, 边/d, ⼴广告/n, ；/w, , 再次/d, 点击/nw, 菜单/n, -, >, 选项/n, ，/w, 在/p, 安全/an, 页⾯面/n, 下/f, 先/d, 将/d, ⼴广告/n, 拦截/v, 设/v, 为/p, 强⼒力/n, ；/w, 再次/d, 打开/v, 链接/nw, 查看/v, ；/w, 切

换/v, 浏览器/n, 模式/n, ，/w, 点击/nw, 刷新/v, 按钮/n, 查看/v, 。/w]

[当/p, 选择/v, 的/uj, ⼴广告/n, 拦截/v, 模式/n, 改变/v, 后/f, ，/w, 拦截/v, 的/uj, ⼴广告/n, 数⽬目/n, 不/d, ⼀一致/a, 。/w]

No. Input 输⼊入

R1 [compatibility/v, mode/n, log/v, click/nw, personal/n, homepage/n, friend/n, gift/n, backword/v, button/n, forwad/v, page/n, content/n]

[兼容/v, 模式/n, 登录/v, 点击/nw, 个⼈人/n, 主页/n, 好友/n, 礼物/n, 后退/v, 按钮/n, 前进/v, 页⾯面/n, 内容/n]

R2 [enter/v, compatibility/v, mode/n, log/v, click/nw, friend/n, link/nw, personal/n, homepage/n, button/n, load/nw, back/v, page/n]

[进⼊入/v, 兼容/v, 模式/n, 登录/v, 点击/nw, 好友/n, 链接/nw, 个⼈人/n, 主页/n, 按钮/n, 加载/nw, 后退/v, 页⾯面/n]

R3 [open/v, IE/n, tool/n, options/n, security/an, ads/n, block/v, select/v, address/n, input/v, page/n, floating/v, corner/nw, not/v]

[打开/v, 浏览器/n, ⼯工具/n, 选项/n, 安全/an, ⼴广告/n, 拦截/v, 选择/v, 地址/n, 输⼊入/v, 页⾯面/n, 悬浮/v, 右下⾓角/nw, 没有/v]

R4 [compatibility/v, mode/n, input/v, groupon/v, click/nw, search/v, button/n, icecream/n, red-bull/nw, back/v, forward/v, back-forward/

nw, result/n]

[兼容/v, 模式/n, 输⼊入/v, 团购/v, 点击/nw, 搜索/v, 按钮/n, 冰淇淋/n, 红⽜牛/nw, 后退/v, 前进/v, 前进后退/nw, 结果/n]

R5 [open/v, IE/n, max/v, condition/n, wait/v, program/n, load/nw, finish/v, do/v, switch/v, full-screen/nw, appear/v, system/n, task/n, missed/v,

possibility/n, mouse/n, thumbnail/n, return/v]

[打开/v, 浏览器/n, 最⼤大化/v, 状况/n, 等待/v, 程序/n, 加载/nw, 完成/v, 进⾏行/v, 切换/v, 全屏/nw, 出现/v, 系统/n, 任务/n, 消失/v, 可

能性/n, ⿏鼠标/n, 略图/n, 恢复/v]

R6 [IE/n, click/nw, menu/n, options/n, security/an, page/n, ads/n, block/v, close/v, open/v, link/nw, page/n, load/nw, find/v, strong/n, check/v,

fail/v]

[浏览器/n, 点击/nw, 菜单/n, 选项/n, 安全/an, 页⾯面/n, ⼴广告/n, 拦截/v, 关闭/v, 打开/v, 链接/nw, ⽹网页/n, 加载/nw, 发现/v, 强⼒力/n, 查

看/v, 失败/v]

R7 [IE/n, click/nw, menu/n, options/n, security/an, page/n, ads/n, block/v, closed/v, open/v, link/nw, appear/v, floating/v, strong/n, check/v,

switch/v, mode/n, refresh/v, button/n, select/v, change/v, number/n]

[浏览器/n, 点击/nw, 菜单/n, 选项/n, 安全/an, 页⾯面/n, ⼴广告/n, 拦截/v, 关闭/v, 打开/v, 链接/nw, 显⽰示/v, 漂浮/v, 强⼒力/n, 查看/v, 切

换/v, 模式/n, 刷新/v, 按钮/n, 选择/v, 改变/v, 数⽬目/n]

Word Freq Word Freq Word Freq Word Freq Word Freq Word Freq进⾏行 1 按钮 4 强⼒力 2 主页 2 改变 1 程序 1

消失 1 输⼊入 2 个⼈人 2 切换 2 出现 1 地址 1

礼物 1 浏览器 4 好友 2 略图 1 内容 1 没有 1

漂浮 1 选项 3 登录 2 前进后退 1 数⽬目 1 查看 2

后退 3 模式 4 搜索 1 拦截 3 右下⾓角 1 可能性 1

冰淇淋 1 打开 4 团购 1 失败 1 关闭 2 点击 5

完成 1 ⼴广告 3 加载 3 菜单 2 进⼊入 1 ⿏鼠标 1

全屏 1 ⼯工具 1 恢复 1 安全 3 最⼤大化 1 显⽰示 1

状况 1 ⽹网页 1 刷新 1 系统 1 任务 1 选择 2

等待 1 链接 3 悬浮 1 页⾯面 5 结果 1 发现 1

前进 2 红⽜牛 1 兼容 3 　 　 　 　 　 　

Word Freq Word Freq Word Freq Word Freq Word Freq Word Freqdo 1 button 4 strong 2 page 2 change 1 program 1

missed 1 input 2 personal 2 switch 2 appear 1 address 1

gift 1 IE 4 friend 2 thumbnail 1 content 1 not 1

floating 1 options 3 log 2 back-forward 1 number 1 check 2

back 3 mode 4 search 1 block 3 corner 1 possibility 1

ice-cream 1 open 4 groupon 1 fail 1 close 2 click 5

finish 1 ads 3 load 3 button 2 enter 1 mouse 1

full-screen 1 tool 1 restore 1 security 3 maximize 1 appear 1

condition 1 page 1 refresh 1 system 1 task 1 select 2

wait 1 link 3 floating 1 page 5 result 1 find 1

forward 2 red-bull 1 compatilbity 3 　 　 　 　 　 　

No. 按钮 强⼒力 主⻚页 输⼊入 个⼈人 切换 浏览器 好友 选项 登录 查看 后退 模式 拦截 打开 关闭 点击 ⼲⼴广告 加载 菜单 安全 选择 链接 ⻚页⾯面 前进 兼容R1 1 0 1 0 1 0 0 1 0 1 0 1 1 0 0 0 1 0 0 0 0 0 0 1 1 1R2 1 0 1 0 1 0 0 1 0 1 0 1 1 0 0 0 1 0 1 0 0 0 1 1 0 1R3 0 0 0 1 0 0 1 0 1 0 0 0 0 1 1 0 0 1 0 0 1 1 0 1 0 0R4 1 0 0 1 0 0 0 0 0 0 0 1 1 0 0 0 1 0 0 0 0 0 0 0 1 1R5 0 0 0 0 0 1 1 0 0 0 0 0 0 0 1 0 0 0 1 0 0 0 0 0 0 0R6 0 1 0 0 0 0 1 0 1 0 1 0 0 1 1 1 1 1 1 1 1 0 1 1 0 0R7 1 1 0 0 0 1 1 0 1 0 1 0 1 1 1 1 1 1 0 1 1 1 1 1 0 0

No. button

strong

homepage

input

personal

switch

IE friend

options login check back mode block open close click

ads load

menu security

select

link page

forward

compatibilityR1 1 0 1 0 1 0 0 1 0 1 0 1 1 0 0 0 1 0 0 0 0 0 0 1 1 1

R2 1 0 1 0 1 0 0 1 0 1 0 1 1 0 0 0 1 0 1 0 0 0 1 1 0 1R3 0 0 0 1 0 0 1 0 1 0 0 0 0 1 1 0 0 1 0 0 1 1 0 1 0 0R4 1 0 0 1 0 0 0 0 0 0 0 1 1 0 0 0 1 0 0 0 0 0 0 0 1 1R5 0 0 0 0 0 1 1 0 0 0 0 0 0 0 1 0 0 0 1 0 0 0 0 0 0 0R6 0 1 0 0 0 0 1 0 1 0 1 0 0 1 1 1 1 1 1 1 1 0 1 1 0 0R7 1 1 0 0 0 1 1 0 1 0 1 0 1 1 1 1 1 1 0 1 1 1 1 1 0 0

No. 按钮 强⼒力 主⻚页 输⼊入 个⼈人 切换 浏览器 好友 选项 登录 查看 后退 模式 拦截 打开 关闭 点击 ⼲⼴广告 加载 菜单 安全 选择 链接 ⻚页⾯面 前进 兼容R1 1 0 1 0 1 0 0 1 0 1 0 1 1 0 0 0 1 0 0 0 0 0 0 1 1 1R2 1 0 1 0 1 0 0 1 0 1 0 1 1 0 0 0 1 0 1 0 0 0 1 1 0 1R3 0 0 0 1 0 0 1 0 1 0 0 0 0 1 1 0 0 1 0 0 1 1 0 1 0 0R4 1 0 0 1 0 0 0 0 0 0 0 1 1 0 0 0 1 0 0 0 0 0 0 0 1 1R5 0 0 0 0 0 1 1 0 0 0 0 0 0 0 1 0 0 0 1 0 0 0 0 0 0 0R6 0 1 0 0 0 0 1 0 1 0 1 0 0 1 1 1 1 1 1 1 1 0 1 1 0 0R7 1 1 0 0 0 1 1 0 1 0 1 0 1 1 1 1 1 1 0 1 1 1 1 1 0 0

No. button

strong

homepage

input

personal

switch

IE friend

options login check back mode block open close click

ads load

menu security

select

link page

forward

compatibilityR1 1 0 1 0 1 0 0 1 0 1 0 1 1 0 0 0 1 0 0 0 0 0 0 1 1 1

R2 1 0 1 0 1 0 0 1 0 1 0 1 1 0 0 0 1 0 1 0 0 0 1 1 0 1R3 0 0 0 1 0 0 1 0 1 0 0 0 0 1 1 0 0 1 0 0 1 1 0 1 0 0R4 1 0 0 1 0 0 0 0 0 0 0 1 1 0 0 0 1 0 0 0 0 0 0 0 1 1R5 0 0 0 0 0 1 1 0 0 0 0 0 0 0 1 0 0 0 1 0 0 0 0 0 0 0R6 0 1 0 0 0 0 1 0 1 0 1 0 0 1 1 1 1 1 1 1 1 0 1 1 0 0R7 1 1 0 0 0 1 1 0 1 0 1 0 1 1 1 1 1 1 0 1 1 1 1 1 0 0

All three prioritization strategies (Div, Risk, and DivRisk) begin with choosing a test report that contains the most keywords (i.e., most information).

Div Prioritization Strategy

• Based on the principle of diversity of test selection

• Greedy algorithm

• Prefer to select the next test report with the maximal distance to already chosen and sorted test-report set

• Fully automatic

Risk Prioritization Strategy• Seeks to reveal faults as early as possible

• Dynamic prioritization strategy, based on thus-far inspection results

• Risk values of each keyword are updated as reports are inspected

• Remaining test reports are sorted based on updated risk values

• Semi-automatic

DivRisk Prioritization Strategy• Hybrid technique to balance the goals of the primary strategies

• Dynamic prioritization strategy, based on thus-far inspection results to assess risk and keyword distance to assess diversity

• Risk values of each keyword are updated (by ẟ) as reports are inspected

• Remaining test reports are sorted based on updated risk values and keyword distance

• First a candidate set (of size nc) of test reports with the greatest distance from the already prioritized set (i.e., diversity), then the report with the greatest risk values (sum of keyword risk values) is selected next

• Semi-automatic

Experiment• Subject: Three Baidu projects from pilot study [Chen, Liu

2014] • Ground truth (report inspection): Baidu developers • Independent variable: Prioritization Strategy

• Div• Risk• DivRisk• Random (baseline)• Best-case• Worst-case

• Dependent variable: APFD • Parameters: ẟ=0.2, nc=8, repeat=50

Results: P1

Results: P1

Results: P2

Results: P2

Results: P3

Results: P3

Observations• Prioritization strategies performed best on P1,

which was the Android mobile application and was dominated by compatibility issues

• Prioritizing is, in general, better than random (i.e., un-prioritized)

• There may be room for improvement to approach the best-case order

Related WorkTest-Suite Prioritization• Prioritizing risky tests

(e.g., Dickenson01, Yan10)

• Diversity prioritization (e.g., Ledru09, Yoo09, Fang14)

Failure-Report Classification• Clustering bug reports

(e.g., Runeson07, Wang08)

• Clustering execution information (e.g., Bowring04, Lo09, Dhaliwal11)

Crowdsourced Testing• E.g., Liu12, Nebeling12,

Pastore13, Dolstra13

Future Work• Images and screenshots may be used in future

test-report prioritization to help distinguish behaviors

• Multi-objective heuristic search could be applied

• Benchmarks of crowdsourced data to be gathered to assist future replication and technique improvement (current archive is at http://www.qingkaishi.com/?Android_Resources_Repository)

Conclusions

• Test-report prioritization technique, based on natural-language descriptions

• Three prioritization strategies, inspired by test-suite prioritization and failure classification

• Results showed promise for our three crowdsourced projects

Test Report Prioritization to Assist Crowdsourced

Testing Zhenyu Chen, Chunrong Fang, Baowen Xu

State Key Laboratory for Novel Software Technology, Nanjing University

Yang Feng, James A. JonesDepartment of Informatics, UC Irvine

This work is partially supported by the National Basic Research Program of China (973 Program 2014CB340702), the National Natural Science Foundation of China (61170067, 61373013), and the

National Science Foundation under awards CAREER CCF-1350837 and CCF-1116943.