2/5/98 UCLA Data Mining Short Course (3) 1

Integrated Data Mining Systems

Wei-Min Shen

Information Sciences Institute

University of Southern California

2/5/98 UCLA Data Mining Short Course (3) 2

Outline

• Objectives for Integrated System

• System Architecture

• Necessary Capabilities

• Representation Languages

• Actual System Descriptions

2/5/98 UCLA Data Mining Short Course (3) 3

Objectives for Integrated KDD Systems

• Carry out the entire KDD process– Data selection

– Data preprocessing

– Data transformation

– Data mining

– Interpretation and evaluation

• Coherently integrate complementary techniques• Amplify human capabilities (e.g. see a lot)• Allow human to control the KDD process

2/5/98 UCLA Data Mining Short Course (3) 4

System Architecture

• Necessary elements– Access to existing data sets or databases– Representation and storage of knowledge– Basic data mining techniques

• Deduction

• Induction

• Visualization

• Use of human guidance

2/5/98 UCLA Data Mining Short Course (3) 5

Deduction

• A rigid inference procedure from the general to the specific– “All computers have CPU” “X is a computer”

“X has CPU”

• Seek evidences for a general hypothesis– “Maybe all computers have CPU”– “Check how many computers in my database

have CPU”

2/5/98 UCLA Data Mining Short Course (3) 6

Induction

• A “not so rigid” inference procedure from the specific to the general– “I drove yesterday,” “you drove yesterday,” “he

drove yesterday,” …...– “every one drove yesterday”

• Seek for general patterns from data

• There are many popular induction methods– Decision trees, rules and lists, NN, ILP, ...

2/5/98 UCLA Data Mining Short Course (3) 7

Visualization

• Allow humans to see very large amounts of data in one visual field

• Provide clues for abstractions by humans

2/5/98 UCLA Data Mining Short Course (3) 8

The Use of Human Guidance

• The need for human guidance– Large amount of data– Large search space for possible patterns– Machines do not human’s intuition yet

• How to encode human knowledge into data mining process?

2/5/98 UCLA Data Mining Short Course (3) 9

Representation

• Languages for data access and manipulation– SQL, Datalog, LDL++, Cobol, C++, …

• Languages for representing knowledge– Prolog, LDL++, Loom, …

• Prefer languages that serve multiple purpose

2/5/98 UCLA Data Mining Short Course (3) 10

Examples of Integrated Systems

• IBM’s Intelligent Miner, Advanced Scout

• Recon

• DBMiner

• DataCrystal

• many more

2/5/98 UCLA Data Mining Short Course (3) 11

Advanced Scout• A system that helps NBA coaches to find

and use patterns hidden in historical game data

• Example patterns– “Glenn Rice played the shooting guard position, he shot

5/6 (83%) on jump shots”

• Widely used by many NBA teams, and coaches say that “it is written with coach in mind”

2/5/98 UCLA Data Mining Short Course (3) 12

Recon

• Inputs: Relational databases

• Outputs: Rule-based models

• Integrate induction, deduction, visualization

2/5/98 UCLA Data Mining Short Course (3) 13

Recon Architecture

Graphical User Interface

Command Module

DeductiveDatabase

RuleInductionVisualization

Recon Server

External DB

Target DBKnowledgeRepository

2/5/98 UCLA Data Mining Short Course (3) 14

Recon Visualization

• Obtain a global view of a data set– a view of tables and columns

• Noticing important phenomena hold on subsets of data– Clusters– Trends– Correlation

2/5/98 UCLA Data Mining Short Course (3) 15

Recon Deductive Database

• Define concepts– high-growth:

• earnings-per-share-growth>50% and dividend-growth>50%

• Allow new concepts to be defined on the existing ones

• Effect: prepare subsets of data for further analysis

2/5/98 UCLA Data Mining Short Course (3) 16

Recon Rule Induction:

• User define target concepts

• Learn a set of rules for the target concepts

• Has heuristics for modifying existing rules

• Example:– If a stock is high-growth at time t, then its

return on investment two quarters later will be greater than 20%

2/5/98 UCLA Data Mining Short Course (3) 17

DBMiner Architecture

Graphical User Interface

Discovery ModuleSQL Server

DatabaseConcept

Hierarchy

2/5/98 UCLA Data Mining Short Course (3) 18

DBMiner Functionalities

• Inputs: Databases and Concept Hierarchy

• Outputs: – Characteristic rules (hypothesis evidence)– Discriminate rules (evidence hypothesis)– Multi-level association rules

2/5/98 UCLA Data Mining Short Course (3) 19

DBMiner Key Idea

• Attribute-Oriented Induction– Organize values of each attribute into a

hierarchy of concepts– Perform rule induction at certain “prime” level

in the hierarchies

• learn rules at a

2/5/98 UCLA Data Mining Short Course (3) 20

DataCrystal (KnowledgeMiner)

• A common-representation language– “Metapatterns”

• An integrated, efficient search engine– “The Discovery Loop”

2/5/98 UCLA Data Mining Short Course (3) 21

Metapatterns• Specifications for type and form of pattern• An example of metapattern

P(X,Y) & Q(Y,Z) R(X,Z)

• Examples of discovered patternscitizen(X,Y) & officialLanguage(Y,Z) speaks(X,Z) [0.98]

parent(X,Y) & ancestor(Y,Z) ancestor(X,Z) [0.99]

• Other MetapatternsIngredients(X, a, b) & Property(X,Y) Cluster(Y)

connects(C,D) & Feature(C,X) & Feature(D,Y) eql(X,Y)

2/5/98 UCLA Data Mining Short Course (3) 22

The Discovery Loop

KnowledgeBase

MetapatternGenerator

DBs

Data

Metapatterns P(X,Y) & Q(Y,Z) R(X,Z)

discovered Patterns

DataQueries

Inductive Actions

Deductive DB

computeStrengthsupervised learningclusteringcase-based reasoningregression analysisvisualization

citizen(X,Y) & officialLanguage(Y,Z) speaks(X,Z)

2/5/98 UCLA Data Mining Short Course (3) 23

DataCrytal Applications• Discover common-sense regularities from a large

knowledge base (MCC)– goodStudent(X,Y), taughtBy(Y,Z) likedBy(X,Z) [0.99]

• Find circuit patterns from a telecommunication database (Bellcore)– connect(X,’cab’,Y,’ept’),endLoc(X,U),loc(Y,V) eql(U,V) [0.98]

• Build prediction models from a chemical research database (Eastman Chemical)– percentage(X,’g306’,Y),density(X,W) F35 (Y,W)

• Construct fault-detection rules from a semiconductor manufacture control database (Motorola)– receipt(W,2),p41(W,Y),time(W,179) allowedVariance(0.9,3.4)

2/5/98 UCLA Data Mining Short Course (3) 24

Metapattern Generation

• Metapatterns are hard to design– A time consuming interactive process

• Challenges– No pre-labeled examples

– No pre-specified concepts

– Mostly relational concepts

– Unsupervised Learning of relational patterns

• So we need to generate metapatterns automatically

2/5/98 UCLA Data Mining Short Course (3) 25

The Algorithm• Inputs: schema, value ranges, thresholds, and

domain knowledge (optional)

• Outputs: relational patterns

• Three main steps– Step 1. Find connections among tables

• relational patterns can only be found among connected tables

– Step 2. Generate transitive metapatterns

• transitive patterns constitute a very interesting subset of relational patterns (implication, inheritence, transfer through, function dependency)

– Step 3. Generate other metapatterns based on previous metapatterns

2/5/98 UCLA Data Mining Short Course (3) 26

Step 1. Find connections• Identify columns that are significantly connected

– two columns are significantly connected if they have the same type and their ranges overlap significantly

– domain knowledge can be used here for • eliminating unnecessary connections (e.g., length, width)• establishing syntactically different connections (e,g., color, frequency)

• Construct the significant connection table (SCT)– a reference name is created for each connected pair– the reference names and the table names are used as

rows and columns of the SCT

2/5/98 UCLA Data Mining Short Course (3) 27

An Abstract DB Example

T1: C11 char(2) C12 integer [1-9] C13 float[0.1-0.9]

T2: C21 integer[11-19] C22 float[0.1-0.9] C23 char(3)

T3: C31 integer[11-19] C32 char(2)

T4: C41 char(3) C42 float[0.0-0.1] C43 integer[1-9]

Schema and value ranges

2/5/98 UCLA Data Mining Short Course (3) 28

Abstract DB Data Tables

C41 C42 C43KKK 0.1 3

SSS 0.0 7OOO 0.0 4

PPP 0.0 5OOO 0.0 5EEE 0.0 4LLL 0.0 6

MMM 0.1 4NNN 0.1 3

SSS 0.0 3QQQ 0.1 7KKK 0.1 6LLL 0.0 6

DDD 0.1 9OOO 0.1 5

C21 C22 C2317 0.6 GGG16 0.8 JJJ15 0.2 NNN16 0.7 PPP13 0.8 TTT11 0.5 KKK14 0.6 CCC13 0.4 KKK12 0.5 OOO14 0.4 OOO

C11 C12 C13MM 8 0.6

TT 4 0.5UU 5 0.7KK 2 0.4LL 9 0.5QQ 5 0.8

JJ 4 0.8MM 5 0.7

JJ 5 0.7OO 5 0.5OO 5 0.9OO 3 0.4

JJ 6 0.2NN 3 0.3

C31 C3215 KK18 LL16 OO18 JJ18 HH16 MM15 KK14 TT16 FF15 LL

T1 T2 T3 T4

2/5/98 UCLA Data Mining Short Course (3) 29

DB Example Continue ...

T1 T2 T3 T4 X1 C13 C22 X2 C11 C32 X3 C12 C43X4 C21 C31 X5 C23 C41

Significant Connection Table

2/5/98 UCLA Data Mining Short Course (3) 30

Step 2: Generate Metapatterns• Convert SCT to a graph G

• Find all predicate cycles in G

• Generate the complete set of transitive metapatterns

2/5/98 UCLA Data Mining Short Course (3) 31

DB Example Continue ...A GrapghG constructed from SCT

T1,X1 T2,X1

T1,X2 T3,X2

T1,X3 T4,X3

T2,X4 T3,X4

T2,X5 T4,X5

2/5/98 UCLA Data Mining Short Course (3) 32

DB Example Continue ...All Predicate Cycls found in G

(T2 X1 X4) (T3 X4 X2) (T1 X2 X1)

(T2 X1 X5) (T4 X5 X3) (T1 X3 X1)

(T2 X5 X1) (T1 X1 X2) (T3 X2 X4) (T2 X4 X5)

(T2 X4 X5) (T4 X5 X3) (T1 X3 X1) (T2 X1 X4)

(T1 X3 X1) (T2 X1 X4) (T3 X4 X2) (T1 X2 X3)

(T3 X2 X4) (T2 X4 X5) (T4 X5 X3) (T1 X3 X2)

(T1 X2 X3) (T4 X3 X5) (T2 X5 X1) (T1 X1 X2)

(T2 X1 X4) (T3 X4 X2) (T1 X2 X3) (T4 X3 X5) (T2 X5 X1)

(T1 X1 X2) (T3 X2 X4) (T2 X4 X5) (T4 X5 X3) (T1 X3 X1)

2/5/98 UCLA Data Mining Short Course (3) 33

DB Example Continue...

• The complete set of metapatternsP1(Y1,Y2) & Q1(Y2,Y3) => R1(Y1,Y3)

P2(Y1,Y2) & Q2(Y2,Y3) & W2(Y3,Y4) => R1(Y1,Y4)

P3(Y1,Y2) & Q3(Y2,Y3) & W3(Y3,Y4) & V3(Y4,Y5) => R3(Y1,Y5)

2/5/98 UCLA Data Mining Short Course (3) 34

Pattern Evaluation• Evaluate each instantiated pattern p of metapattern P by

– Computing two values:

• strength: ps = prob(R | L,U,I) = (|R|+1) / (|L| + 2)

• base: pb = sqrt( (1- ps) ps / N )

– Comparing with specified thresholds s and b:

if pb < b,

then if (ps > s) or (ps < (1-s))

then accept p

else mark p as plausible

else discard p

2/5/98 UCLA Data Mining Short Course (3) 35

Examples of Evaluation

accept(T2 X4 X1) (T3 X4 X2) (T1 X2 X3) => (T1 X3 X1) [0.8, 0.15](T1 X2 X1) (T3 X4 X2) (T2 X4 X5) (T4 X5 X3) => (T1 X3 X1) [0.9, 0.11]

plausible(T1 X2 X3) (T4 X5 X3) (T2 X4 X5) => (T3 X4 X2) [0.5, 0.14]

discard(T3 X4 X2) (T2 X4 X5) (T4 X5 X3) => (T1 X2 X3) [0.4, 0.9]

when s=0.8, and b=0.5

2/5/98 UCLA Data Mining Short Course (3) 36

Step 3. Propose More Metapatterns• For each metapattern P that has many plausible

patterns, do

– Select a (meta)constraint C and append it to the left hand side of P

• C must connect to at least one predicate in P

• C is a build-in predicate (e.g., =)

• C is suggested by the domain knowledge

• An ExampleP1(Y1,Y2) & Q1(Y2,Y3) & S1(Y2,O) => R1(Y1,Y3)

2/5/98 UCLA Data Mining Short Course (3) 37

A Small Network Example

0

3 6

5

7

841

2

2/5/98 UCLA Data Mining Short Course (3) 38

Network Data Tables

a1 a20 10 20 30 40 50 60 81 23 23 43 53 63 84 54 64 86 87 67 8

b1 b20 10 31 23 23 44 54 66 87 67 8

Can-reach Linked-to

2/5/98 UCLA Data Mining Short Course (3) 39

Network Example Continue ...Schema and Value Ranges

CAN-REACH: A1 integer[0-8] A2 integer[0-8]LINKED-TO: B1 integer[0-8] B2 integer[0-8]

CAN-REACH LINKED-TOX1 A1 B1 X2 A2 B1 X3 A2 B2

Significant Connection Table

2/5/98 UCLA Data Mining Short Course (3) 40

Network Example Continue ...The SCT Graph

CR, X1 LT, X1

CR, X2 LT, X2

CR, X3 LT, X3

2/5/98 UCLA Data Mining Short Course (3) 41

Network Example Continue ...All Predicate Cycles

(LINKED-TO X1 X3) (CAN-REACH X1 X3)(LINKED-TO X3 X1) (CAN-REACH X1 X2) (LINKED-TO X2 X3)(CAN-REACH X1 X2) (LINKED-TO X2 X3) (CAN-REACH X3 X1)

Evaluate against DB (LINKED-TO X1 X3) => (CAN-REACH X1 X3) [1.0, 10]

(CAN-REACH X1 X2) (LINKED-TO X2 X3) => (CAN-REACH X1 X3) [1.0, 11]

(CAN-REACH X1 X2) (CAN-REACH X1 X3) => (LINKED-TO X2 X3) [0.1, 89](CAN-REACH X1 X3) (LINKED-TO X2 X3) => (CAN-REACH X1 X2) [0.4, 31]

(CAN-REACH X1 X3) => (LINKED-TO X1 X3) [0.5, 19]

2/5/98 UCLA Data Mining Short Course (3) 42

Characteristics of Metapattern Generation

• Unsupervised learning of relational (transitivity) patterns– with no pre-specify concepts– with no pre-label examples– that have probabilistic significance – directly from databases