Text Mining: Finding Nuggets in Mountains of Textual Data Jochen Dijrre, Peter Gerstl, Roland...
-
date post
22-Dec-2015 -
Category
Documents
-
view
216 -
download
2
Transcript of Text Mining: Finding Nuggets in Mountains of Textual Data Jochen Dijrre, Peter Gerstl, Roland...
Text Mining: Finding Nuggets in Mountains of Textual Data
Jochen Dijrre, Peter Gerstl, Roland Seiffert
Presented by Huimin Ye
Outline
Motivation Methodology Feature Extraction Clustering and Categorizing Some Applications Comparison with Data Mining Conclusion & Exam Questions
Motivation
A large portion of a company’s data is unstructured or semi-structured
Letters Emails Phone recordings Contracts
Technical documents Patents Web pages Articles
Definition
Text Mining: the discovery by computer of new, previously
unknown information, by automatically extracting information from different written resources
Typical Applications
Summarizing documents Discovering/monitoring relations among people,
places, organizations, etc Customer profile analysis Trend analysis Documents summarization Spam Identification Public health early warning Event tracks
Outline
Motivation Methodology Comparison with Data Mining Feature Extraction Clustering and Categorizing Some Applications Conclusion & Exam Questions
Methodology: Challenges
Information is in unstructured textual form Natural language interpretation is difficult &
complex task! (not fully possible) Text mining deals with huge collections of
documents
Methodology: Two Aspects
Knowledge Discovery Extraction of codified information Mining proper; determining some structure
Information Distillation Analysis of feature distribution
Two Text Mining Approaches
Extraction Extraction of codified information from single
document
Analysis Analysis of the features to detect patterns, trends,
etc, over whole collections of documents
Outline
Motivation Methodology Feature Extraction Clustering and Categorizing Some Applications Comparison with Data Mining Conclusion & Exam Questions
IBM Intelligent Miner for Text
IBM introduced Intelligent Miner for Text in 1998
SDK with: Feature extraction, clustering, categorization, and more
Traditional components (search engine, etc) The rest of the paper describes text mining
methodology of Intelligent Miner.
Feature Extraction
Recognize and classify “significant” vocabulary items from the text
Categories of vocabulary Proper names Multiword terms Abbreviations Relations Other useful things: numerical forms of
numbers, percentages, money, etc
Canonical Form Examples
Normalize numbers, money Four = 4, five-hundred dollar = $500
Conversion of date to normal form Morphological variants
Drive, drove, driven = drive Proper names and other forms
Mr. Johnson, Bob Johnson, The author = Bob Johnson
Feature Extraction Approach
Linguistically motivated heuristics Pattern matching Limited lexical information (part-of-speech) Avoid analyzing with too much depth
Does not use too much lexical information No in-depth syntactic or semantic analysis
Advantages to IBM’s approach
Processing is very fast (helps when dealing with huge amounts of data)
Heuristics work reasonably well Generally applicable to any domain
Outline
Motivation Methodology Comparison with Data Mining Feature Extraction Clustering and Categorizing Some Applications Conclusion & Exam Questions
Clustering
Fully automatic process Documents are grouped according to
similarity of their feature vectors Each cluster is labeled by a listing of the
common terms/keywords Good for getting an overview of a document
collection
Two Clustering Engines
Hierarchical clustering Orders the clusters into a tree reflecting various
levels of similarity
Binary relational clustering Flat clustering Relationships of different strengths between
clusters, reflecting similarity
Clustering Model
Categorization
Assigns documents to preexisting categories Classes of documents are defined by providing a set
of sample documents. Training phase produces “categorization schema” Documents can be assigned to more than one
category If confidence is low, document is set aside for
human intervention
Categorization Model
Outline
Motivation Methodology Feature Extraction Clustering and Categorizing Some Applications Comparison with Data Mining Conclusion & Exam Questions
Applications
Customer Relationship Management application provided by IBM Intelligent Miner for Text called “Customer Relationship Intelligence” “Help companies better understand what their
customers want and what they think about the company itself”
Customer Intelligence Process
Take as input body of communications with customer
Cluster the documents to identify issues Characterize the clusters to identify the
conditions for problems Assign new messages to appropriate clusters
Customer Intelligence Usage
Knowledge Discovery Clustering used to create a structure that can be
interpreted Information Distillation
Refinement and extension of clustering results Interpreting the resultsTuning of the clustering processSelecting meaningful clusters
Outline
Motivation Methodology Feature Extraction Clustering and Categorizing Some Applications Comparison with Data Mining Conclusion & Exam Questions
Comparison with Data Mining
Data mining Discover hidden
models. tries to generalize all
of the data into a single model.
marketing, medicine, health care
Text mining Discover hidden
facts. tries to understand
the details, cross reference between individual instances
biosciences, customer profile analysis
Conclusion
This paper introduced text mining and how it differs from data mining proper.
Focused on the tasks of feature extraction and clustering/categorization
Presented an overview of the tools/methods of IBM’s Intelligent Miner for Text
Exam Question #1
Name an example of each of the two main classes of applications of text mining. Knowledge Discovery: Discovering a common
customer complaint in a large collection of documents containing customer feedback.
Information Distillation: Filtering future comments into pre-defined categories
Exam Question #2
How does the procedure for text mining differ from the procedure for data mining? Adds feature extraction phase Infeasible for humans to select features manually The feature vectors are, in general, highly
dimensional and sparse
Exam Question #3
In the Nominator program of IBM’s Intelligent Miner for Text, an objective of the design is to enable rapid extraction of names from large amounts of text. How does this decision affect the ability of the program to interpret the semantics of text? Does not perform in-depth syntactic or semantic analysis
of the text; the results are fast but only heuristic with regards to actual semantics of the text.
Questions?