Retrieving Relevant Reports from a Customer Engagement Repository Dharmesh Thakkar Zhen Ming Jiang...
-
Upload
alvin-hudson -
Category
Documents
-
view
214 -
download
1
Transcript of Retrieving Relevant Reports from a Customer Engagement Repository Dharmesh Thakkar Zhen Ming Jiang...
Retrieving Relevant Reports from a Customer Engagement Repository
Dharmesh Thakkar
Zhen Ming Jiang
Ahmed E. HassanSchool of Computing, Queen’s University, Canada
Gilbert Hamann
Parminder FloraResearch In Motion (RIM), Canada
Software Maintenance: Customer Support
ApplicationSupportAnalyst
SymptomsIdentified Problems
SolutionsExecution
Logs
Customer Engagement
Report
Customer Engagement Repository
Create Store
Contains
Attempted Workarounds
Retrieving Relevant Reports
■ State of Practice:– No systematic techniques to retrieve and use
information for future engagements– Keyword searching is limited:
• depends on the search skills and experience of the analyst and peculiarity of the problem
Customer SupportProblem Statement
■ We want to find customers with similar operational and problem profiles
■ We can reuse prior solutions and knowledge
Heavy Email, Light Web, Light MDS Light Email, Light Web, Light MDS Light Email, Heavy Web, Light MDS Heavy Email, Heavy Web, No MDS Light Email, Light Web, Heavy MDS
Other Customers
CompareNew
Customer Engagement
Using Logs forCustomer Support
■ Execution logs are readily available and contain – Operational Profile: usage patterns (heavy
users of email from device, or to device, or light users of calendar, etc.)
– Signature Profile: specific error line patterns (connection timeout, database limits, messages queued up, etc.)
■ Find the most similar profile
Execution Logs
■ Contain time-stamped sequence ofevents at runtime
■ Readily available representatives of both feature executions and problems
<time> Queuing new mail msgid=ABC threadid=XYZ<time> Instant message. Sending packet to client msgid=ABC threadid=XYZ<time> New meeting request msgid=ABC threadid=XYZ<time> Client established IMAP session emailid=ABC threadid=XYZ<time> Client disconnected. Cannot deliver msgid=ABC threadid=XYZ<time> New contact in address book emailid=ABC threadid=XYZ<time> User initiated appointment deletion emailid=ABC threadid=XYZ
Example
010002000
C2
0
2000
4000
C1
0
2000
4000
C3
MTH
MTH m
ore
requ
ests
MFH
View C
alend
ar
Mee
ting
Reque
st
Appoin
tmen
t syn
c
Mes
sage
mov
e
Conta
cts
HTTPM
DS
0500
1000150020002500
NewSupport Request
Other Customers
Compare
Our Technique
CustomerExecution
Logs
Convert Log Lines to Event Distribution
Compare Event
Distributions
Customer Engagement Repository
Compare Event
Distributions
Identify Signature
Events
Signature Event Distribution
OUTPUT RESULT SET
Closest Customer Engagement Reports
wrtSignature Profile
Closest Customer Engagement Reports
wrtOperational Profile
Event Distribution
Log Lines to Event Distribution
■ Remove dynamic information– Example: Given the two log lines “Open inbox
user=A” and “Open inbox user=B”, map both lines to the event “Open inbox user=?”
■ Use event percentages to compare event logs for different running lengths without bias
CustomerExecution
Logs
Convert Log Lines to Event Distribution
Compare Event
Distributions
Customer Engagement Repository
Compare Event
Distributions
Identify Rare Signature
Events
Rare Signature Event
Distribution
OUTPUT RESULT SET
Closest Customer Engagement Reports
wrtRare Event Profile
Closest Customer Engagement Reports
wrtOperational Profile
Event Distribution
CompareEvent Distributions
CustomerExecution
Logs
Convert Log Lines to Event Distribution
Compare Event
Distributions
Customer Engagement Repository
Compare Event
Distributions
Identify Rare Signature
Events
Rare Signature Event
Distribution
OUTPUT RESULT SET
Closest Customer Engagement Reports
wrtRare Event Profile
Closest Customer Engagement Reports
wrtOperational Profile
Event Distribution
D1 D2 D3
■ Kullback-Leibler Divergence
■ Cosine Distance
Identify Signature Events
■ Signature Events have a different frequency when compared to events in other log files– Example signature events: dropped
connections, thread dumps, and full queues
■ Chi-square test identifies such events
CustomerExecution
Logs
Convert Log Lines to Event Distribution
Compare Event
Distributions
Customer Engagement Repository
Compare Event
Distributions
Identify Rare Signature
Events
Rare Signature Event
Distribution
OUTPUT RESULT SET
Closest Customer Engagement Reports
wrtRare Event Profile
Closest Customer Engagement Reports
wrtOperational Profile
Event Distribution
Measuring PerformanceC
RelevantLog Files
RRetrievedLog Files
F2
F1
F3F4
F5
CR
■ Precision = 2/4 = 50%100% precise if all the retrieved log files are relevant
■ Recall = 2/3 = 67%100% recall if all the relevant log files are retrieved
The Big Picture
CustomerExecution
Logs
Convert Log Lines to Event
Distribution
Compare Event
Distributions
Customer Engagement Repository
Compare Event
Distributions
Identify Signuare Events
Signature Event Distribution
OUTPUT RESULT SET
Closest Customer Engagement Reports
wrtSignature Profile
Closest Customer Engagement Reports
wrtOperational Profile
Event Distribution
Case Studies
■ Case Study I– Dell DVD Store open source application– Code instrumentation done for event logging– Built the execution log repository by applying
synthetic workloads, changing the workload parameters each time
■ Case Study II– Globally deployed commercial application– More than 500 unique execution events
Case Study Results■ Dell DVD Store
– 100% precision and recall on both operational profile based and signature profile based retrieval
■ Commercial Application– 100% precision and recall for signature profile based
retrieval– Results for operational profile based retrieval:
ExperimentCount of Log Files
K-L Distance Cosine Distance
Precision Recall Precision Recall
Single Feature Group 28 67.71% 90.28% 67.71% 90.28%Multiple Feature Groups 28 60.71% 80.95% 75.00% 100.00%All Feature Groups 12 72.92% 97.22% 62.50% 83.33%Real World Log Files 12 54.17% 72.22% 68.75% 91.67%All the Log Files 80 59.93% 79.90% 56.72% 75.62%
Sources of Errors
■ Events that do not correspond directly to a particular operational feature, such as idle time events, server health check events, startup and shutdown events
■ Imbalance in the event logging
Imbalance in Event Logging
F1E2
E1
F2E4
E3
E6
E5
F3E8
E7
E10
E9
400
F1E2
E1
F2E4
E3
E6
E5
F3E8
E7
E10
E9
F1E2
E1
F2E4
E3
E6
E5
F3E8
E7
E10
E9
E12
E11
E12
E11
E12
E11
OP1 OP2 OP3
400
200
400
450
200
400
400
220
Related Work
■ Data mining techniques on textual information[Hui and Jha, 2000]– Cons: Limited results, depending on analyst’s search skills and
peculiarity of the problem
■ Using customer usage data [Elbaum and Narla, 2004]– Cons: Customer usage data rarely exists
■ Clustering HTTP execution logs [Menascé, 1999]– Cons: Complex process, works only for HTTP logs
■ Software Agent Deployment to build operational profile [Ramanujam et. al., 2006]– Cons: Intrusive, complex, costly
Conclusion