Expertise Finding for Question Answering (QA) ServicesOctober 16, 2014Department of Knowledge Service EngineeringProf. Jae-Gil Lee

210/16/2014

Brief Bio• Currently, an associate professor at Department of

Knowledge Service Engineering, KAIST• Homepage: http://dm.kaist.ac.kr/jaegil

• Lab homepage: http://dm.kaist.ac.kr/

• Previously, worked at IBM Almaden Research Center and University of Illinois at Urbana-Champaign

• Areas of Interest: Data Mining and Big Data

310/16/2014

Table of Contents• Community-based Question Answering (CQA) Services• Background and Motivation

• Methodology Overview

• Evaluation Results

• Social Search Engines for Location-Based Questions• Background and Motivation

• System Architecture and User Interface

• Evaluation Results

410/16/2014

Question Answering (QA) Services

QA services are good at Recently updated information Personalized information Advice & opinion[Budalakoti et al., 2010]

Questions Answers KnowledgeBase

Search

Experts

510/16/2014

Community-based Question Answering (CQA) Services

Naver Knowledge-In Yahoo! Answers

50,000 questions per day 160,000 questions per day

610/16/2014

Motivation of Our Study• Most contributions (i.e.,

answers) in CQA services are made by a small number of heavy users

• Recently-joined users are prone to leave CQA ser-vices very soon

Only 8.4% of answerers remained after a year

Making the long tail stay longer before they leave is of prime importance towards the success of the services

710/16/2014

Problem Setting• To whom does the service provider need to pay special at-

tention? Recently-joined (i.e., light) users who are likely to become contributive (i.e., heavy) users

• Goal: estimating the likelihood of a light user becoming a heavy user (mainly by his/her expertise)

• Challenges: lack of information about the light user

어장관리 ?

810/16/2014

Intuition behind Our Methodology• A person’s active vocabulary reveals his/her

knowledge

• Vocabulary has sharable characteristics so that domain-specific words are repeatedly used by expert answerers

SSD

NAND

ECC

RAM

Device

Memory

Computer

NAND

ECC

RAMSSD

Operation

Data

Drive

Q&A 1 by Answerer 1 Q&A 2 by Answerer 2

Domain-SpecificVocabularies

CommonVocabularies

LevelDifference

SharableCharacteristics

910/16/2014

Estimated Expertise

Heavy Users Words Light Users

The more expert a user is, the higher the level of words he/she used is.

1010/16/2014

Availability• Simply measuring the number of a user’s answers with

their importance proportional to their recency

1110/16/2014

Answer Affordance• Being defined as the likelihood of a light user becom-

ing a heavy user if he/she is treated specially

• Considering both expertise and availability

𝐴𝑓𝑓𝑜𝑟𝑑𝑎𝑛𝑐𝑒 (𝑢𝑙 )=¿

1210/16/2014

Data Set• Collected from Naver Knowledge-In (KiN, 지식인 )

• Spanning ten years (from Sept. 2002 to Aug. 2012)

• Including two categories: Computers and Travel• Computers: factual information, Travel: subjective opinions

• The entropy was used for measuring the expertise of a user, working well especially for the categories where factual exper-tise is primarily sought after [Adamic et al., 2008]

• StatisticsComputers Travel

# of answers 3,926,794 585,316

# of words 191,502 232,076

# of users 228,369 44,866

1310/16/2014

Evaluation Setting (1/2)• Finding the top-k users by

Affordance() for light users our methodology

• Retrieving the top-k directoryexperts managed by KiN competitor

• Measuring the two measuresfor the next one month• User availability: the ratio of the number of the top-k users who

appeared on the day to the total number of users who appeared on that day

• Answer possession: the ratio of the number of the answers posted by the top-k users on the day to the total number of answers posted on that day

1410/16/2014

Evaluation Setting (2/2)

Ten year period

Sept. 2002 July 2011 July 2012 Aug. 2012

Used for deriving the word levels Used for finding top-k experts by our methodology

Picked up the top-k directory experts managed by KiN

Monitored the user availability and answer possession

1510/16/2014

The result of the answer possession

The result of the user availability (a) Computers (b) Travel

(a) Computers (b) Travel

top-400 top-200

top-400 top-200

See the paper for the technical details.

Sung, J., Lee, J., and Lee, U., "Booming Up the Long Tails: Discovering Potentially Contributive Users in Community-Based Question Answering Services," In Proc. 7th Int'l AAAI Conf. on Weblogs and Social Media (ICWSM), Cambridge, Massachusetts, July 2013.

This paper received the Best Paper Award at AAAI ICWSM-13.

1710/16/2014

Table of Contents• Community-based Question Answering (CQA) Services• Background and Motivation

• Methodology Overview

• Evaluation Results

• Social Search Engines for Location-Based Questions• Background and Motivation

• System Architecture and User Interface

• Evaluation Results

1810/16/2014

Social Search (1/2)• A new paradigm of knowledge acquisition that relies

on the people of a questioner’s social network

1910/16/2014

Social Search (2/2)

If you want to get some opinions or advices from your online friends, what do you do?

Not knowing whom to ask Knowing whom to ask

Taking advantage of both approaches

Social Search

2010/16/2014

KiN Here ( 지식인 위치질문 )• A query is routed by finding a match between a target

location of a query and a relevant location of a user

동 단위로 추가

2110/16/2014

Location-Based Questions• Informally defined as “search for a business or place of in-

terest that is tied to a specific geographical location”[Amin et al., 2009]

• Very popular especially in mobile search and typically sub-jective• Mobile search is estimated to comprise 10% 30% of all searches ∼• About 9 10% of the queries from Yahoo! mobile search∼ ,

over 15% of 1 million Google queries from PDA devices , and about 10% of 10 million Bing mobile queries were identified as location-based questions

• In a set of location-based questions, 63% of them were non-factual, and the remaining 37% of them were factual

Mobile social search is the best way to process location-based questions

2210/16/2014

Glaucus: A Social Search Engine for Location-Based Questions

1. Asking a question to Glaucus2. Selecting proper experts3. Routing the question to the experts4. Returning an answer to the questioner5. (Optional) Rating the answer

GlaucusSocial Search

Engine

User Database

1: Query

Users

2: Selected Experts

3: Query

Answer 4: Answer

5: Feedback

Crawling

Questioner

2310/16/2014

User Interface• An Android app has been developed and is under

(closed) beta testing

Questioner Answerer

2410/16/2014

Data Collection• Being able to collect who visited where and when on

geosocial networking services such as Foursquare• Users check-in to a venue and also may leave a tip

• Our crawler collects such information upon user approval

2510/16/2014

Expert Finding

Venue

Location

Category

Time

Misc.

Venue

Location

Category

Time

Misc.

Location Aspect Model

Questioner

Question

Other Users

Online Friend?

SimilarityCalculation

Score

Score

Score

Score

Top-k

2610/16/2014

Evaluation Setting• Collected check-in’s and tips from Foursquare

(foursquare.com)

• Confined to the places in the Gangnam District

• Ranging from April 2012 to December 2012

• Statistics Variable Value

# of users 9,163

# of places (venues) 1,220

# of check-in’s 243,114

# of tips 40,248

2710/16/2014

Evaluation Results

0123456789

10SocialTelescopeAardvark Glaucus

DC

G

Set 1 Set 2 Set 3

3.94 3.994.07

6.61 6.31 6.68

8.25 8.827.78

Experts Non-Experts1

2

3

Ans

wer

Rat

ing 2.37

1.97

Qualification of the Experts:Two human judges investigated the profiles of the experts selected by the three systems for 30 questions (distributed to 3 sets) and gave a score in 3 scales.

Quality of the Answers:Two human judges examined the quality of the answers―both from experts and non-experts―and gave a score in 3 scales.

2810/16/2014

Mobile User Availability• Motivation

• Study MethodologyContext

Smart Phone Log

External Information(Time, Date)

AvailabilityClassifier

Decision Tree,SVM,

Random Forest …

26Features

Class LabelClassification

Model

Availability?

Training Prediction26

Features

Availability

2910/16/2014

User Behavior Collection

분류 데이터 종류 수집 방법

스마트폰 Context

Data

배터리 정보 ( 배터리 잔량 , 충전 여부 , 충전 모드 )

백그라운드 수집

전화 정보 ( 통화 시작시간 , 통화 소요시간 , 수신/ 발신 여부 )

메시지 정보 ( 문자 시간 , 수신 / 발신 여부 )

GPS 정보 ( 위도 , 경도 )기기 정보 ( 진동모드 , 무음모드 , 비행기모드 ,

CPU 사용량 , 헤드폰모드 , 스크린 점등 )

주위 정보 ( 주변 조명 밝기 , 주변 소음 세기 )WIFI 정보 (WIFI On/Off, SSID, 신호 세기 )

Cellular 정보 (Cellular On/Off, 신호 세기 )

애플리케이션 정보 ( 애플리케이션 이름 , 애플리케이션 구동 시간 )

가용성Data

특정 시각에서의 응답 가능 여부 직접 입력

3010/16/2014

Preliminary Evaluation Results• Accuracy• 10-fold cross validation

• 10 users for 5 weeks

• Important Features• 1st: Time, Day of Week

• 2nd: Running Apps

• 3rd: WIFI SSID, # of Apps (30 mins), Time of Day

Model Accuracy

Baseline (Always Available) 0.53

Naïve Bayesian 0.66

SVM 0.64

KNN 0.62

Decision Tree 0.64

Adaboost 0.61

Random Forest 0.7

time_h

our

time_w

eekNum

app_pkg

wifi_ss

id

prev_a

pps

time_sl

ot

loc_lab

bat_lev

cell_

strn

stat_c

puill_

lev

bat_plug

prev_k

akaos

prev_lo

cs0

1

2

3

4

5

6

7

8

9

10

See the paper for the technical details.

Choy, M., Lee, J., Gweon, G., and Kim, D., "Glaucus: Exploiting the Wisdom of Crowds for Location-Based Queries in Mobile Environments," In Proc. 8th Int'l AAAI Conf. on Weblogs and Social Media (ICWSM), Ann Arbor, Michigan, June 2014.

Thank you very much!Any Questions?E-mail: jaegil@kaist.ac.kr Homepage: http://dm.kaist.ac.kr/