[IEEE The 9th IEEE International Conference on E-Commerce Technology and The 4th IEEE International...

A Conversation-Mining System for Gathering Insights to Improve AgentProductivity

Hironori Takeuchi�, L Venkata Subramaniam�, Tetsuya Nasukawa�,Shourya Roy�and Sreeram Balakrishnan�

�IBM Research, Tokyo Research Laboratory �IBM Research, India Research LaboratoryShimotsuruma 1623-14, Yamato-shi Institutional Area 4, Block-C, Vasant Kunj

Kanagawa 2428502 Japan New Delhi 110070 India�hironori, nasukawa�@jp.ibm.com �lvsubram, rshourya, srbalakr�@in.ibm.com

Abstract

We describe a method to analyze transcripts of conver-sations between customers and agents in a contact center.The aim is to obtain actionable insights from the conver-sations to improve agent performance. Our approach hasthree steps. First we segment the call into logical parts.Next we extract relevant phrases within different segments.Finally we do two dimensional association analysis to iden-tify actionable trends. We use real data from a contact cen-ter to identify specific actions by agents that result in posi-tive outcomes. We also show that implementing the action-able results in improved agent productivity.

1 Introduction

Many companies today operate contact centers to sellproducts and services, handle customer issues and addressproduct-related and services-related issues. Contact centersemploy a handful of people, generally known as QualityAnalysts (QAs), to analyze the agent-customer interactions.These analysts randomly sample calls and listen to the con-versations between agents and customers. The analysts rateagents based on these calls on various metrics such as com-munication skills, problem identification, problem resolu-tion, etc. The analysts also categorize calls into differentcategories based on the nature of the problems or queries.Examples of domain-independent categories could be rudeagent behavior, delayed response, whereas, categories in aspecific domain such as technical support could be installa-tion issue, network outage, etc. This kind of manual anal-ysis suffers from some serious problems. Most importantamong them being lack of consistency or subjectivity be-tween different QAs. Quite often one analyst may mark aconversation as showing an agent lacking proper commu-

nication skills while another analyst sees the same data asshowing a resolution problem. Secondly only a small frac-tion of the calls can be analyzed manually, which introducessignificant sampling bias in the analysis. Also, the manualanalysis process is slow - with calls as long as 30 minutes,depending on the nature of the business. Last but definitelynot the least, the manual analysis process can be quite shal-low in nature and more often than not the analysis results inreports showing the whats but not the whys.

Text Analytics can play a vital role in improving the ef-ficiency of this process as well as doing deeper and moreinsightful analysis. However application of text analytics isnot very common in contact centers because contact centersdo not have in-house text analytics capabilities. Also owingto the confidential nature of data, the business informationcontained in emails, call logs, and call transcripts can not bemade public. Hence such data never reaches the text miningcommunity. We have been interacting with services organi-zations including contact centers and in this paper we willshare our experiences of applying text analytics techniqueson really “real-life” data.

In this paper we propose techniques using which wecould quickly identify business insights that domain expertscould not find in spite of their spending days analyzing thesame data. We show that associations between agent utter-ances and outcomes can be found if we segment the callsand aggregate phrases within the call segments. In partic-ular we analyze calls to answer some of the key questionsfaced by QAs with the objective of improving agent produc-tivity.

� Is there an association between agent and customer ut-terances and call outcomes?

� Can some specific agent utterances boost the numberof positive outcomes?

The 9th IEEE International Conference on E-CommerceTechnology and The 4th IEEE International Conferenceon Enterprise Computing, E-Commerce and E-Services(CEC-EEE 2007)0-7695-2913-5/07 $25.00 © 2007

��

Today within a contact center there are a variety of tex-tual information sources. There is data, of the interactionsbetween customers and agents, in the form of summariesof contact records by agents, online chat transcripts, phoneconversation transcripts, emails, and SMS.

There has been a lot of work on specific tools for con-tact centers. These include call type classification for thepurpose of categorizing calls [6], call routing [2], obtainingcall log summaries [1], agent assisting and monitoring [3],building of domain models [5], analyzing records of cus-tomer contacts [4].

��

In this paper, we try to analyze aggregated informationfrom a large set of calls to a contact center rather than fo-cusing on the specific information in each call. We de-velop text analytics techniques to identify specific agent ut-terances that have an impact on call outcomes. We presenttechniques to automatically segment the conversations intological segments and then collect the phrases within the seg-ments to find associations that impact the call outcome. Wealso show that these actionable insights when fed back tothe agents result in improved performance.

2 Description of the Dataset

We collected calls from a car rental help desk and manu-ally transcribed them. Figure 1 shows part of a call that hasbeen transcribed. The calls comprise of customer and agentturns. There are three types of calls:

1. Booked Call: Calls in which a car is booked. Bookedcalls can get picked-up or not picked-up.

2. Unbooked Call: Customer didn’t make a booking.3. Service Call: Enquiries, changes and cancellations.In a car rental conversation, each conversation consists

of the following segments, opening, customer requirements,offering car and rate by agent, getting personal details, re-cap, mandatory enquiries and closing. These segments arecommonly found in reservation calls. For unbooked or ser-vice calls, some of the segments might be missing.

Because the call process is defined clearly, all reserva-tion calls look similar in spite of having different results (interms of pickup). Finding the differences in the conversa-tions that affect the outcomes is the key to the analysis.

3 System for Analyzing Transcribed Calls

Our system consists of three main components, sentenceboundary detection and call segmentation, information ex-traction, and association analysis.

AGENT: Welcome to CarCompanyA. My name is Albert. How may I help you?

.........

AGENT: Allright may i know the location you want to pick the car from.

CUSTOMER: Aah ok I need it from SFO.

AGENT: For what date and time.

.........

AGENT : alright mam let me recap the dates you want to pick it up from SFO

on 3rd August and drop it off on august 6th in LA alright

CUSTOMER : and one more questions Is it just in states or could you travel

out of states

.........

CUSTOMER : ok ok Thank you

AGENT : Thank you for calling CarCompanyA and you have a great day good bye

Figure 1. Transcript of a Car Rental Dialog(part of a call)

��

Call transcription when done manually is tedious and er-ror prone. One error transcribers make is missing out oninserting punctuation marks. However, many applicationssuch as information extraction and natural language pro-cessing benefit from (or even require) a sentence structure.We use the partially labeled set from the manual transcrip-tions to learn and improve the sentence boundaries by iden-tifying phrases which typically appear at the beginning orend of a sentence. The sentence boundaries, help in theconcept extraction phase that follows.

Calls to a contact center typically follow a predeterminedcall flow where a sequence of events or actions happen. Ex-amples of such actions could be an agent gathering pickupand return details from the customer, the agent enquiringwhether the customer has a valid driver license. Hence acall can be broken up into segments based on the particularaction being performed in that part of the call. For exam-ple all calls contain a greeting segment and a conclusionsegment. Depending on the domain they may also containsegments for asking customer requirements, mandatory en-quiries, etc. Typical segments found in a car rental processwere mentioned in section 2. Like sentence boundary detec-tion, call segmentation helps in the concept extraction phaseby making identification of concepts easier. In many casescorrect information extraction depends on the context of theconcept being extracted. The phrase credit card may getmentioned in many parts of a call - a customer asking theagent if payment by a particular credit card is acceptable,the agent asking for credit card details from the customer,the agent verifying credit card details from the customer,etc. An easy way to check if the agent has verified creditcard details is to look for the phrase credit card in the recapsegment.


Call segmentation is done using a supervised techniquewhere a subset of the calls is manually segmented. Fromthis set of manually segmented documents we extract twosets of keywords for each segment:

1. Frequent keywords obtained by taking the trigrams andbigrams with the highest frequency in each segment.Unigrams are avoided because most of the high fre-quency words are stop words (like, the, is etc).

2. Discriminating keywords obtained by taking the ratioof the frequent phrases (unigrams, bigrams and tri-grams) in a particular segment to their frequency in thewhole corpus with preference being given to trigrams.

For each segment the top 10 keywords are chosen as rep-resentative keywords. Also the probability of a particularsegment type following another segment type is learnt fromthe labeled training set. For new calls, each speaker turnis tagged as belonging to a particular segment based on thekeywords present in it and the information about the seg-ment to which the preceding turn belongs.

�� !"�� #� � ��

We make a list of words extracted from call transcrip-tions sorted by their frequency and assign semantic cat-egories to words that are considered important using thedomain expert’s knowledge. In this process appropriatecanonical forms are assigned to take care of phrasal vari-ations. These mapping rules are registered in a dictionaryas follows.

� child seat [noun]�� child seat [vehicle feature]

� master card [noun]�� credit card [payment methods]

In natural language, there are many ways to express thesame concept. We therefore extract concepts using user de-fined patterns of grammatical forms and expressions. Forexample to identify how car rental agents are putting theirrequests or mentioning value selling phrases, following pat-terns for extracting phrases are defined.

� please + VERB� VERB[request]

� just + NUMERIC + dollars � mention of goodrate[value selling]

We combine these extracted keywords and expressionswith the corresponding segment and speaker information.For example, when we find “make a reservation” in the cus-tomer’s turn in the opening segment, we consider the ex-pression as “customer intent at start of call”.

�� $�� $��

Once the appropriate concepts have been extracted fromthe documents, we look for associations between concepts

to extract insights. For example, we can fill in each cellin the two-dimensional Table 1 by counting the number oftexts that contain both the column and row concepts. Thisallows us to find any strong associations between vehicletypes and locations.

Table 1. 2-Dimensional Association AnalysisVehicle type category

SUV mid-size full-size luxury carLocation New Yorkcategory Los Angeles

SeattleBoston

4 Analysis and Results

%�� $�� $��

We collected recorded calls from the car rental help deskand manually transcribed them. The dataset has been de-scribed in Section 2. We used 137 booked calls and con-structed a text mining system for the difference analysis“picked-up” vs. “not picked-up”.

We first identify the key concepts from the call tran-scriptions and group them under appropriate semantic cat-egories. We hypothesize that there are two types of cus-tomer intentions at the start of call, booking customer andrates customer depending on their willingness to make abooking, and such customer intention can be changed bythe agent’s actions. Under this classification, we preparedthe following semantic categories:

� Customer intention at start of call: From the cus-tomer’s first or second utterance, we extract the fol-lowing intentions based on the patterns.

– booking customer: would like to make a book-ing, need to pickup a car, � � �

– rates customer: can I check the rates, I would liketo know the rates for vans, � � �

� Discount-relating phrases: discount, corporate pro-gram, motor club, � � � are registered into the domaindictionary.

� Value selling phrases: we extract phrases mentioninggood rate and good vehicle by matching patterns.

– mention of good rate: wonderful rate, savemoney, just need to pay this low amount, � � �

– mention of good vehicle: good car, fantastic car,latest model, � � �

Using these categories, we tried to find insights to im-prove agent productivity.

%�� $��

Table 2 shows the result of two dimensional associationanalysis between customer types and pickup results for 137


Table 2. Association between CustomerTypes and Pick Up Results

Customer types extracted from texts Pick up resultbased on customer intent at start of call pickup not picked upbooking customer(w/ strong start)(70) ��

rates customer(w/ weak start) (37) ��

reservation calls. From these results, we see that 67% (47out of 70) of the booking customers picked up the reservedcar and only 35% (13 out of 37) of the rates customerspicked it up. This shows that the pickup result can be pre-dicted from the customer’s first or second utterance. Wenow look for actions by agents that convert rates customersto “pickup”.

Table 3 shows how mentioning discount-related phrasesaffects the pickup results for rates customer and book-ing customer. Table 4 shows how mentioning a good rate ora good vehicle affects the pickup results. From these tables,

Table 3. Association between Mention ofDiscount-Related Phrases and Pick Up Re-sults

�� Pick up resultMention of discount-related phrases by agents pickup not picked up

yes(21) ��

no(16) � ��

�� Pick up resultMention of discount-related phrases by agents pickup not picked up

yes(40) ��

no(30) ��

Table 4. Association between Mention ofValue Selling Phrases and Pick Up Results

�� Pick up informationMention of value selling phrases by agents pickup not picked up

yes(17) � �

no(20) � ��

�� Pick up informationMention of value selling phrases by agents pickup not picked up

yes(28) ��

no(42) ��

it can be seen that mentioning discount-related phrases andvalue selling phrases affects the outcome in both the book-ing customer and rates customer cases. In particular, for arates customer, the probability that the booked car will bepicked up, � (pick-up) is improved to 0.476 by mentioningdiscount-related phrases.

As results, we derived the following actionable insights.� There are two types of customers in reservation calls.

– Booking customer tends to pickup the reservedcar.

– Rates customer tends not to pickup a reservedcar.

� In the rates customer case, the “pickup” ratio is im-proved by mentioning discount-related phrases.

� In both the rates customer and booking customercase, the “pickup” ratio is improved by mentioningvalue selling phrases.

%�� &�� $� � ��&��

By implementing the actionable insights derived fromthe analysis, we verified improvements in pickup. We di-vided 83 agents in the reservation center into two groups.One of them, consisting of 22 agents, was trained on theinsights from the text analysis. The remaining 61 agentswere not told about these findings. We compared these twogroups over a period of one month to see how the insightscontributed to improving agent performance. As the evalu-ation metric, we used the pickup ratio - that is the ratio ofthe number of “pickups” to the number of reservations.

Following the training the pickup ratio of the trainedagents increased by 4.75%. The average pickup ratio for theremaining agents increased by 2.08%. Before training theratios of both groups were comparable. Considering this,it can be estimated that by implementing the actionable in-sights the pickup ratio was improved by about 2.67%. Weconfirmed that this difference is meaningful because a t-testrejects the possibility of an accidental difference betweenthem at a probability level of �=0.1 (p-value=0.0674).

5 Conclusions and Future Work

In this paper, we have proposed methods for analyz-ing conversational data to improve agent productivity. Weshowed that certain phrases if used by agents result in pos-itive call outcomes. As a part of the future work we wouldlike analyze calls transcribed using automatic speech recog-nition technology to get insights.

References

[1] S. Douglas, D. Agarwal, T. Alonso, R. M. Bell, M. Gilbert,D. F. Swayne, and C. Volinsky. Mining customer care dialogsfor “daily news”. IEEE Transaction on Speech and AudioProcessing, 13(5):652–660, 2005.

[2] H.-K. J. Kuo and C.-H. Lee. Discriminative training of naturallanguage call routers. IEEE Transaction on Speech and AudioProcessing, 11(1):24–35, 2003.

[3] G. Mishne, D. Carmel, R. Hoory, A. Roytman, and A. Sof-fer. Automatic analysis of call-center conversations. In Pro-ceedings of ACM Conference on Information and KnowledgeManagement (CIKM), pages 453–459, 2005.

[4] T. Nasukawa and T. Nagano. Text analysis and knowledgemining system. IBM Systems Journal, pages 967–984, 2001.

[5] S. Roy and L. V. Subramaniam. Automatic generation of do-main models for call centers from noisy transcriptions. InProceedings of the 21st International Conference on Com-putational Linguistics and 44th Annual Meeting of the ACL(COLING/ACL), pages 737–744, 2006.

[6] M. Tang, B. Pellom, and K. Hacioglu. Call-type classificationand unsupervised training for the call center domain. In Pro-ceesings of IEEE Workshop on Automatic Speech Recognitionand Understanding, pages 204–208, 2003.


[IEEE The 9th IEEE International Conference on E-Commerce Technology and The 4th IEEE International...

Documents

Transcript of [IEEE The 9th IEEE International Conference on E-Commerce Technology and The 4th IEEE International...