2
Overview Text Mining Review Converting Unstructured Text to Structured Data Segmenting Textual Data Demonstrations
Text Mining – Why and How
The volume of text data is much greater than that of numeric data
The means dealing with text data is far from enough
5
What Text Mining Is Text mining is a process that employs a set of
algorithms for converting unstructured text into structured data objects and the quantitative methods used to analyze these data objects.
“SAS defines text mining as the process of investigating a large collection of free-form documents in order to discover and use the knowledge that exists in the collection as a whole.” (SAS Text Miner: Distilling Textual Data for Competitive Business Advantage)
6
What Text Mining Is Not
Text mining is not a text summarization tool an information extraction methodology a natural language processor.
8
The SAS Text Mining Process1. Preprocess document files to create a SAS data set.
TMFILTER macro SAS language features
2. Parse the document field. PARSE tab in Text Miner Stemming Part-of-speech tagging Entities Stop/start lists Synonym lists And so forth
continued...
9
The SAS Text Mining Process3. Derive the term by document frequency matrix.
The Text Miner Transform tab Frequency weights Term weights
4. Transform the term by document frequency matrix. The Text Miner Transform Tab Singular Value Decomposition (SVD) Roll Up Terms
5. Perform the analysis. Exploration Clustering/unsupervised learning Predictive modeling
10
Text Mining Strengths Clustering documents in a corpus Investigating word (token) distribution across
documents within a corpus Identifying words with the highest discriminatory
power Classifying documents into predefined
categories Integrating text data with structured data to
enrich predictive modeling endeavors
11
Text Mining Deficiencies
Text mining algorithms perform poorly in distinguishing negations, for example: Herman was involved in a motor vehicle accident. Herman was NOT involved in a motor vehicle accident.
Text mining cannot generally make value judgments, for example, classifying an article as positive or negative with respect to any tokens it contains.
12
Text Mining Deficiencies
Text mining algorithms do not work well with large documents. Performance is slow. Increased term occurrence across documents
decreases separation of documents.
13
The SAS Text Mining Process1. Preprocess document files to create a SAS data set.
TMFILTER macro SAS language features
2. Parse the document field. PARSE tab in Text Miner Stemming Part-of-speech tagging Entities Stop/start lists Synonym lists And so forth
continued...
14
The SAS Text Mining Process3. Derive the term by document frequency matrix.
The Text Miner Transform tab Frequency weights Term weights
4. Transform the term by document frequency matrix. The Text Miner Transform Tab Singular Value Decomposition (SVD) Roll Up Terms
5. Perform the analysis. Exploration Clustering/unsupervised learning Predictive modeling
15
This demonstration illustrates how to use the TMFILTER macro to process groups of text files.
Using the TMFILTER Macro with the Newsgroups Data
16
SAS Text Miner Text Processing Features
Text parsing Removal of stop words Part-of-speech tagging Stems and synonym handling Entities
17
Stop Words Stop words are words that have little or no value in
identifying a document or in comparing documents. Standard stop lists contain stop words that are
Articles (the, a, this) Conjunctions (and, but, or) Prepositions (of, from, by).
Custom stop lists identify low information words, like the word “computer” in a collection of articles about computers.
19
Default Stop Lists A default stop list or a user-defined stop list defines
stop words to be removed.
Default stop lists: English: sashelp.stoplst French: sashelp.frchstop German: sashelp.grmnstop Mixed: sashelp.mixdstop
20
Stop List versus Start List Use a start list when
– documents are dominated by “technical jargon”– domain expertise can enhance text mining.
Use a stop list when– documents are loosely related: news, business
reports, Internet searches– domain expertise is not available.
21
Issues in Creating a Start List Do not just add high frequency terms.
– Low frequency terms that only appear in a few documents may be good discriminators.
– High frequency terms may be candidates for a stop list.
Data-derived start lists should be reviewed by domain experts.
22
Tagging Parts of Speech Determines if the word is a common noun, verb,
adjective, proper noun, adverb, and so forth. Disambiguate parts of speech when a word is used in a
different context, I wish that my bank had more ATM machines. You can bank on either Philadelphia or Oakland
winning the Super Bowl next year. Settlers living on the west bank of the river were
forced to relocate.
25
Stemming May employ algorithm and/or table look up
– Porter stemmer– Levin stemmer
Errors of commission (organizationorgan) Errors of omission (matricesmatrix) Can be related to spell checking
continued...
26
Stemming Examples
BIG: BIG, BIGGER, BIGGEST REACH: REACH, REACHES, REACHED, REACHING WORK: WORK, WORKS, WORKED, WORKING
CHILD: CHILD, CHILDREN KNIFE: KNIFE, KNIVES
PERRO: PERRO, PERRA (Spanish, male and female dog)
28
Stemming in Text Miner Text Miner performs stemming to derive stem
synonyms, for example, run/ran/runs/running, and combines these with defined synonyms, for example, run/sprint.
The default synonym data set for Text Miner, sashelp.engsynms, is primarily for illustration.
Synonyms may split based on part of speech, for example, teach/train=verb, locomotive/train=noun.
29
Synonyms Language dictionaries Technical jargon Abbreviations Specialty dictionaries
Note: This could be associated with stemming in file preprocessing.
continued...
31
Synonym Lists Default: sashelp.engsynms (ToolsSettings) User Defined
– SAS Data Set– Three fields: TERM ($25.), PARENT ($25.),
CATEGORY ($12.)– Example: TERM=EM, PARENT=Enterprise Miner,
CATEGORY=PRODUCT
continued...
33
Term-Document Frequency Matrices
Term ID D1 D2 Dn…
T1 1 D1,1 D1,2 D1,n…
T2 2 D2,1 D2,2 D2,n…
… …
Di,j=count of number of times word i occurs in document j
Documents
34
Term-Document Frequency Matrices
Pitfalls Sparse cells (many zeroes) Weak discriminatory power Too large
Solution Term frequency functions Singular value decomposition
35
Weighted Term-Document Frequency Matrix
Word phrase ID D1 D2 Dn…
T1 1 W1,1 W1,2 W1,n…
T2 2 W2,1 W2,2 W2,n…
… …
Wi,j=weight associated with word i in document j
36
Frequency and Term Weights Notation
jia ji document in appears that termfrequency ,
collection
documentin appears that termfrequency igi
collection in the documents ofnumber nappears ermin which t documents ofnumber idi
i
jiji ga
p ,,
37
Deriving the Weighted Term-Document Frequency Matrix
ijiij LGa ˆ
frequency expectedˆ ija
weighttermiG
weightfrequency ijL
38
Transformed Term-Document Frequency Matrix Elements
The original frequencies
in the Term-Document Frequency Matrix are transformed to the “expected” frequencies
jia ,
ijiji GLa ,,ˆ
39
Default Weights
j iji
i
ijijj
ijiji ag
g
ap
n
ppG ,,
)(log
)(log1
2
2
)1(log2 ijij aL
Term Weight=Entropy
Frequency Weight=Log
40
Frequency Weights
)1(log2 ijij aLLog
Binary
otherwise 0
document in is termif 1 jiLij
Noneijij aL
41
Singular Value Decomposition
Classical SVD in statistics: A=UV For term-document frequency matrix A, U is the matrix of term
vectors, is a diagonal matrix with singular values along the diagonal, and V is the matrix of document vectors.
The projection V* is output as a set of SVD dimensions for each document, with the dimensions stored in the variables COL1, COL2, and so forth. V* is a sub-matrix of V determined by the maximum dimension specified by the user.
Resolution (low/medium/high) changes the cutoff value for selecting a “significant” number of dimensions.
42
SVD is very useful for
Compression Noise reduction Finding ”concepts” or ”topics” (text mining/LSI) Data exploration and visualizing data (e.g. spatial
data/PCA) Classification (of e.g. handwritten digits)
43
SVD appears under different names
Principal Component Analysis (PCA) Latent Semantic Indexing (LSI)/Latent Semantic
Analysis (LSA) Karhunen-Loeve expansion/Hotelling transform (in
image processing)
45
The Text Mining Project
Document analysis is the goal of the project– Exploratory analysis of document collections– Clustering of documents as an aid to human
evaluation of documents
46
Text Mining as Part of a Data Mining Project
Predictive modeling with many fields, one or more of which are unstructured text
Recommender systems Others
Precision vs. Recall
Measure to describe how effective a binary text classifier predicts documents that are relevant to a particular category.
Precision – The percentage of the predicted positive in all positive instances
– Precision = TP / (TP + FN) Recall – How well the classifier can find relevant documents and
properly assign them to their correct category– Recall = TP / (TP + FP)
47
48
Text Mining as Part of a Data Mining Project
The goals of the project influence how text mining is performed.
A single unstructured text field becomes a set of K quantitative inputs.
Memory-Based Reasoning
Memory-based reasoning is a process that identifies similar cases and applies the information that is obtained from these cases to a new record. In Enterprise Miner, the Memory-Based Reasoning node is a modeling tool that uses a k-nearest neighbor algorithm to categorize or predict observations.
The k-nearest neighbor algorithm takes a data set and a probe, where each observation in the data set is composed of a set of variables and the probe has one value for each variable. The distance between an observation and the probe is calculated. The k observations that have the smallest distances to the probe are the k-nearest neighbor to that probe.
49
Top Related