Data Mining Toon Calders

Data Mining

Toon Calders

Explosive Growth of Data: from terabytes to petabytes

– Data collection and data availability

– Major sources of abundant data

Why Data mining?

Why Data mining?

We are drowning in data, but starving for knowledge!

“Necessity is the mother of invention”—Data mining—

Automated analysis of massive data sets

0

500,000

1,000,000

1,500,000

2,000,000

2,500,000

3,000,000

3,500,000

4,000,000

1995 1996 1997 1998 1999

The Data Gap

Total new disk (TB) since 1995

Number of analysts

What Is Data Mining?

Data mining (knowledge discovery from data) – Extraction of interesting (non-trivial, implicit, previously

unknown and potentially useful) patterns or knowledge from

huge amount of data

Alternative names– Knowledge discovery (mining) in databases (KDD), knowledge

extraction, data/pattern analysis, data archeology, data dredging, information harvesting, business intelligence, etc.

Data analysis and decision support

– Market analysis and management

– Risk analysis and management

– Fraud detection and detection of unusual patterns (outliers)

Other Applications

– Text mining (news group, email, documents) and Web mining

– Stream data mining

– Bioinformatics and bio-data analysis

Current Applications

Process mining can be used for:– Process discovery (What is the process?)

– Delta analysis (Are we doing what was specified?)

– Performance analysis (How can we improve?)

process mining

Registerorder

Prepareshipment

Shipgoods

Receivepayment

(Re)sendbill

Contactcustomer

Archiveorder

Ex. 3: Process Mining

Ex. 3: Process Mining

case 1 : task A case 2 : task A case 3 : task A case 3 : task B case 1 : task B case 1 : task C case 2 : task C case 4 : task A case 2 : task B case 2 : task D case 5 : task E case 4 : task C case 1 : task D case 3 : task C case 3 : task D case 4 : task B case 5 : task F case 4 : task D

A

B

C

D

E F

Data Mining Tasks

Previous lectures:– Classification [Predictive]

– Clustering [Descriptive]

This lecture:– Association Rule Discovery [Descriptive]

– Sequential Pattern Discovery [Descriptive]

Other techniques:– Regression [Predictive]

– Deviation Detection [Predictive]

Outline of today’s lecture

Association Rule Mining– Frequent itemsets and association rules

– Algorithms: Apriori and Eclat

Sequential Pattern Mining– Mining frequent episodes

– Algorithms: WinEpi and MinEpi

Other types of patterns– strings, graphs, …

– process mining

Association Rule Mining

Definition– Frequent itemsets

– Association rules

Frequent itemset mining– breadth-first Apriori

– depth-first Eclat



Given a set of transactions, find rules that will predict the occurrence of an item based on the occurrences of other items in the transaction

Market-Basket transactions

TID Items

1 Bread, Milk

2 Bread, Diaper, Beer, Eggs

3 Milk, Diaper, Beer, Coke

4 Bread, Milk, Diaper, Beer

5 Bread, Milk, Diaper, Coke

Example of Association Rules

{Diaper} {Beer},{Milk, Bread} {Eggs,Coke},{Beer, Bread} {Milk},

Implication means co-occurrence, not causality!

Definition: Frequent Itemset

Itemset– A collection of one or more items

Example: {Milk, Bread, Diaper}

– k-itemset An itemset that contains k items

Support count ()– Frequency of occurrence of an itemset

– E.g. ({Milk, Bread,Diaper}) = 2

Support– Fraction of transactions that contain an

itemset

– E.g. s({Milk, Bread, Diaper}) = 2/5

Frequent Itemset– An itemset whose support is greater

than or equal to a minsup threshold

TID Items

1 Bread, Milk





Definition: Association Rule

Example:Beer}Diaper,Milk{

4.052

|T|)BeerDiaper,,Milk(

s

67.032

)Diaper,Milk()BeerDiaper,Milk,(

c

Association Rule– An implication expression of the form

X Y, where X and Y are itemsets

– Example: {Milk, Diaper} {Beer}

Rule Evaluation Metrics– Support (s)

Fraction of transactions that contain both X and Y

– Confidence (c) Measures how often items in Y

appear in transactions thatcontain X

TID Items

1 Bread, Milk





Association Rule Mining Task

Given a set of transactions T, the goal of association rule mining is to find all rules having – support ≥ minsup threshold

– confidence ≥ minconf threshold

Brute-force approach:– List all possible association rules

– Compute the support and confidence for each rule

– Prune rules that fail the minsup and minconf thresholds

Computationally prohibitive!

Mining Association Rules

Example of Rules:

{Milk,Diaper} {Beer} (s=0.4, c=0.67){Milk,Beer} {Diaper} (s=0.4, c=1.0){Diaper,Beer} {Milk} (s=0.4, c=0.67){Beer} {Milk,Diaper} (s=0.4, c=0.67) {Diaper} {Milk,Beer} (s=0.4, c=0.5) {Milk} {Diaper,Beer} (s=0.4, c=0.5)

TID Items

1 Bread, Milk





Observations:

• All the above rules are binary partitions of the same itemset: {Milk, Diaper, Beer}

• Rules originating from the same itemset have identical support but can have different confidence

• Thus, we may decouple the support and confidence requirements

Mining Association Rules

Two-step approach: 1. Frequent Itemset Generation

– Generate all itemsets whose support minsup

2. Rule Generation– Generate high confidence rules from each frequent itemset,

where each rule is a binary partitioning of a frequent itemset

Frequent itemset generation is still computationally expensive

Frequent Itemset Generation

null

AB AC AD AE BC BD BE CD CE DE

A B C D E

ABC ABD ABE ACD ACE ADE BCD BCE BDE CDE

ABCD ABCE ABDE ACDE BCDE

ABCDE

Given d items, there are 2d possible candidate itemsets

Frequent Itemset Generation

Brute-force approach: – Each itemset in the lattice is a candidate frequent itemset

– Count the support of each candidate by scanning the database

– Match each transaction against every candidate

– Complexity ~ O(NMw) => Expensive since M = 2d !!!

TID Items 1 Bread, Milk 2 Bread, Diaper, Beer, Eggs 3 Milk, Diaper, Beer, Coke 4 Bread, Milk, Diaper, Beer 5 Bread, Milk, Diaper, Coke

N

Transactions List ofCandidates

M

w

Frequent Itemset Generation Strategies

Reduce the number of candidates (M)– Complete search: M=2d

– Use pruning techniques to reduce M

Reduce the number of transactions (N)– Reduce size of N as the size of itemset increases– Used by DHP and vertical-based mining algorithms

Reduce the number of comparisons (NM)– Use efficient data structures to store the candidates or

transactions– No need to match every candidate against every

transaction

Reducing Number of Candidates

Apriori principle:– If an itemset is frequent, then all of its subsets must also

be frequent

Apriori principle holds due to the following property of the support measure:

– Support of an itemset never exceeds the support of its subsets

– This is known as the anti-monotone property of support

)()()(:, YsXsYXYX

Found to be Infrequent

null


A B C D E



ABCDE

Illustrating Apriori Principle

null


A B C D E



ABCDEPruned supersets

Illustrating Apriori Principle

Item CountBread 4Coke 2Milk 4Beer 3Diaper 4Eggs 1

Itemset Count{Bread,Milk} 3{Bread,Beer} 2{Bread,Diaper} 3{Milk,Beer} 2{Milk,Diaper} 3{Beer,Diaper} 3

Itemset Count {Bread,Milk,Diaper} 3

Items (1-itemsets)

Pairs (2-itemsets)

(No need to generatecandidates involving Cokeor Eggs)

Triplets (3-itemsets)Minimum Support = 3

If every subset is considered, 6C1 + 6C2 + 6C3 = 41

With support-based pruning,6 + 6 + 1 = 13

Apriori

A CB D

{}

minsup=2

0 0 0 0

Candidates

1 B, C2 B, C3 A, C, D4 A, B, C, D5 B, D

Apriori

A CB D

{}

0 1 1 0

minsup=2

1 B, C2 B, C3 A, C, D4 A, B, C, D5 B, D

Candidates

Apriori

A CB D

{}

0 2 2 0

minsup=2

1 B, C2 B, C3 A, C, D4 A, B, C, D5 B, D

Candidates

Apriori

A CB D

{}

1 2 3 1

minsup=2

1 B, C2 B, C3 A, C, D4 A, B, C, D5 B, D

Candidates

Apriori

A CB D

{}

2 3 4 2

minsup=2

1 B, C2 B, C3 A, C, D4 A, B, C, D5 B, D

Candidates

Apriori

A CB D

{}

2 4 4 3

minsup=2

1 B, C2 B, C3 A, C, D4 A, B, C, D5 B, D

Candidates

Apriori

AB BCAC AD CDBD

A CB D

{}

2 4 4 3

Candidates

minsup=2

1 B, C2 B, C3 A, C, D4 A, B, C, D5 B, D

Apriori

AB BCAC AD CDBD

A CB D

{}

2 4 4 3

1 2 2 3 2 2

minsup=2

1 B, C2 B, C3 A, C, D4 A, B, C, D5 B, D

Apriori

ACD BCD

AB BCAC AD CDBD

A CB D

{}

1 2 2 3 2 2

Candidates

2 4 4 3

minsup=2

1 B, C2 B, C3 A, C, D4 A, B, C, D5 B, D

Apriori

ACD BCD

AB BCAC AD CDBD

A CB D

{}

1 2 2 3 2 2

2 1

2 4 4 3

minsup=2

1 B, C2 B, C3 A, C, D4 A, B, C, D5 B, D

Apriori Algorithm

Apriori Algorithm:

k := 1

C1 := { {A} | A is an item}

Repeat until Ck = {}

Count the support of each candidate in Ck

– in one scan over DB

Fk := { I Ck : I is frequent}

Generate new candidates

Ck+1 := { I : |I| = k+1 and all J I with |J|=k are in Fk}

k:=k+1

Return i=1…k-1 Fi

Depth-first strategy

Recursive procedure– FSET(DB) = frequent sets in DB

Based on divide-and-conquer– Count frequency of all items

let D be a frequent item

– FSET(DB) =

Frequent sets with item D +

Frequent sets without item D


Frequent items– A, B, C, D

Frequent sets with D:– remove transactions without D

and D itself from DB

– Count frequent sets: A, B, C, AC

– Append D: AD, BD, CD, ACD

Frequent sets without D:– remove D from all transactions in DB

– Find frequent sets: AC, BC

1 B, C2 B, C3 A, C, D4 A, B, C, D5 B, D

1 B, C2 B, C3 A, C, D4 A, B, C, D5 B, D

1 B, C2 B, C3 A, C, D4 A, B, C, D5 B, D

Depth-First Algorithm

1 B, C2 B, C3 A, C, D4 A, B, C, D5 B, D

DBminsup=2


1 B, C2 B, C3 A, C, D4 A, B, C, D5 B, D

A: 2B: 4C: 4D: 3

DBminsup=2


1 B, C2 B, C3 A, C, D4 A, B, C, D5 B, D

A: 2B: 4C: 4D: 3

3 A, C4 A, B, C5 B,

A: 2B: 2C: 2

DBDB[D]

minsup=2


1 B, C2 B, C3 A, C, D4 A, B, C, D5 B, D

A: 2B: 4C: 4D: 3

3 A, C4 A, B, C5 B,

A: 2B: 2C: 2

3 A, 4 A, B

A: 2

DBDB[D]

DB[CD]

minsup=2


1 B, C2 B, C3 A, C, D4 A, B, C, D5 B, D

A: 2B: 4C: 4D: 3

3 A, C4 A, B, C5 B,

A: 2B: 2C: 2

3 A, 4 A, B

A: 2

DBDB[D]

DB[CD]

AC: 2

minsup=2


1 B, C2 B, C3 A, C, D4 A, B, C, D5 B, D

A: 2B: 4C: 4D: 3

3 A, C4 A, B, C5 B,

A: 2B: 2C: 2

DBDB[D]

AC: 2

minsup=2


1 B, C2 B, C3 A, C, D4 A, B, C, D5 B, D

A: 2B: 4C: 4D: 3

3 A, C4 A, B, C5 B,

A: 2B: 2C: 2

DBDB[D]

AC: 2

4 A

DB[BD]

A:1

minsup=2


1 B, C2 B, C3 A, C, D4 A, B, C, D5 B, D

A: 2B: 4C: 4D: 3

3 A, C4 A, B, C5 B,

A: 2B: 2C: 2

DBDB[D]

AC: 2

minsup=2


1 B, C2 B, C3 A, C, D4 A, B, C, D5 B, D

A: 2B: 4C: 4D: 3

3 A, C4 A, B, C5 B,

A: 2B: 2C: 2

DBDB[D]

AC: 2

AD: 2BD: 2CD: 2ACD: 2

minsup=2


1 B, C2 B, C3 A, C, D4 A, B, C, D5 B, D

A: 2B: 4C: 4D: 3

DB


minsup=2


1 B, C2 B, C3 A, C, D4 A, B, C, D5 B, D

A: 2B: 4C: 4D: 3

DB


1 B2 B3 A4 A, B

DB[C]

A: 2B: 3

minsup=2


1 B, C2 B, C3 A, C, D4 A, B, C, D5 B, D

A: 2B: 4C: 4D: 3

DB


1 B2 B3 A4 A, B

DB[C]

A: 2B: 3

1

2

4 AA: 1

DB[BC]minsup=2


1 B, C2 B, C3 A, C, D4 A, B, C, D5 B, D

A: 2B: 4C: 4D: 3

DB


1 B2 B3 A4 A, B

DB[C]

A: 2B: 3

minsup=2


1 B, C2 B, C3 A, C, D4 A, B, C, D5 B, D

A: 2B: 4C: 4D: 3

DB

AD: 2BD: 2CD: 2ACD: 2AC: 2BC: 3

1 B2 B3 A4 A, B

DB[C]

A: 2B: 3

minsup=2


1 B, C2 B, C3 A, C, D4 A, B, C, D5 B, D

A: 2B: 4C: 4D: 3

DB


minsup=2


1 B, C2 B, C3 A, C, D4 A, B, C, D5 B, D

A: 2B: 4C: 4D: 3

DB


1 24 A 5

DB[B]

A:1

minsup=2


1 B, C2 B, C3 A, C, D4 A, B, C, D5 B, D

A: 2B: 4C: 4D: 3

DB


minsup=2


1 B, C2 B, C3 A, C, D4 A, B, C, D5 B, D

A: 2B: 4C: 4D: 3

DB


Final set of frequent itemsets

minsup=2


FSET(DB):

1. Count frequency of items in DB

2. F := { A | A is frequent in DB }

3. // Remove infrequent items from DB

DB := { T F : TDB }

4. For all frequent items D except last one do:

// Find frequent, strict supersets of {D} in DB:

4a. Let DB[D] := { T \ {D} | T DB, D T }

4b. F := F { (I D) : I in FSET(DB[D]) }

4c. // Remove D from DB

DB := { T \ {D} : TDB }

5. Return F


All depth-first algorithms use this strategy Difference = data structure for DB

– prefix-tree: FPGrowth

– vertical database: Eclat

ECLAT

For each item, store a list of transaction ids (tids)

TID Items1 A,B,E2 B,C,D3 C,E4 A,C,D5 A,B,C,D6 A,E7 A,B8 A,B,C9 A,C,D

10 B

HorizontalData Layout

A B C D E1 1 2 2 14 2 3 4 35 5 4 5 66 7 8 97 8 98 109

Vertical Data Layout

TID-list

ECLAT

Support of item A = length of its tidlist Remove item A from DB: remove tidlist of A Create conditional database DB[E]:

– Intersect all other tidlists with the tidlist of E

– Only keep frequent items

A B C D E 1 1 2 2 1 4 2 3 4 3 5 5 4 5 6 6 7 8 9 7 8 9 8 10 9

A B C D 1 1 3 6

A B C 1 1 3 6


Remember:– original problem: find rules XY such that

support(XY) minsupsupport(XY) / support(X) minconf

– Frequent itemsets = the combinations XY

Hence:– Get XY by splitting up the frequent itemsets I

Rule Generation

Given a frequent itemset L, find all non-empty subsets f L such that f L – f satisfies the minimum confidence requirement– If {A,B,C,D} is a frequent itemset, candidate rules:

ABC D, ABD C, ACD B, BCD A, A BCD, B ACD, C ABD, D ABCAB CD, AC BD, AD BC, BC AD, BD AC, CD AB,

If |L| = k, then there are 2k – 2 candidate association rules (ignoring L and L)

Rule Generation

How to efficiently generate rules from frequent itemsets?– In general, confidence does not have an anti-monotone

propertyc(ABC D) can be larger or smaller than c(AB

D)

– But confidence of rules generated from the same itemset has an anti-monotone property

– e.g., L = {A,B,C,D}:

c(ABC D) c(AB CD) c(A BCD) Confidence is anti-monotone w.r.t. number of items on the RHS of the rule

Rule Generation for Apriori Algorithm

ABCD=>{ }

BCD=>A ACD=>B ABD=>C ABC=>D

BC=>ADBD=>ACCD=>AB AD=>BC AC=>BD AB=>CD

D=>ABC C=>ABD B=>ACD A=>BCD

Lattice of rulesABCD=>{ }

BCD=>A ACD=>B ABD=>C ABC=>D

BC=>ADBD=>ACCD=>AB AD=>BC AC=>BD AB=>CD

D=>ABC C=>ABD B=>ACD A=>BCD

Pruned Rules

Low Confidence Rule

Summary: Association Rule Mining

Find associations X Y– rule appears in sufficient large part of the database– conditional probability P(Y | X) is high

This problem can be split into two sub-problems:– find frequent itemsets– split frequent itemsets to get association rules

Finding frequent itemsets:– Apriori-property– breadth-first vs depth-first algorithms

From itemsets to association rules– split up frequent sets, use anti-monotonicity

Outline






– process mining

In many applications, the order and transaction times are very important:– stock prices

– events in a networking environmentcrash, starting a program, certain commands

Specific format of the data is very important

Goal: find “temporal rules”, order is important.

Series and Sequences

Example– 70 % of the customers that buy shoes and socks, will

buy shoe polish within 5 days.

– User U1 logging on, followed by User U2 starting program P, is always followed by a crash.

Here, we will concentate on the problem of finding frequent episodes– can be used in the same way as itemsets

– split episodes to get the rules

Series and Sequences

Event sequence: sequence of pairs (e,t), e is an event, t an integer indicating the time of occurrence of e.

An linear episode is a sequence of events

<e1, …, en>.

A window of length w is an interval [s,e] with

(e-s+1) = w. An episode E=<e1, …, en> occurs in sequence

S=<(s1,t1), …, (sm,tm)> within window W=[s,e] if there exist integers s i1 < … < in e such that for all j=1…n, (ej,ij) is in S.

Episode Mining

Episode mining: support measure

Given a sequence SFind all linear episodes that occur frequently in S

Episode mining: support measure

Given a sequence SFind all linear episodes that occur frequently in S

Given an integer w. The w-support of an episode E=<e1, …, en> in a sequence S=<(s1,t1), …, (sm,tm)> is the number of windows W of length w such that E occurs in S within window W.

Note: If an episode occurs in a very short time span, it will be in many subsequent windows, and thus contribute a lot to the support count!

Example

S = 

E = 

E occurs in S within window [0,4], within [1,4], within [5,9], …

The 5-support of E in S is 3, since E is only in the following

windows of length 5: [0,4], [1,5], [5,9]

b a a c b a a b c

An episode E1=<e1, …, en> is a sub-episode of E2=<f1,…,fm>, denoted E1 E2 if there exist integers 1 i1 < … < in m such that for all j=1…n, ej=fij.

Example< b, a, a, c > is a sub-episode of <a, b, c, a, a, b, c>.

Episode Mining Problem

Given a sequence w, a minimal support minsup, and a window width w, find all episodes that have a w-support above minsup.

Monotonicity

Let S be a sequence, E1, E2 episodes, w a number.

If E1 E2, then the w-support(E2) w-support(E1).

WinEpi Algorithm

We can again apply a level-wise algorithm like Apriori.

Start with small episodes, only proceed with a larger episode if all sub-episodes are frequent.<a,a,b> is evaluated after <a>, , <a,a>, <a,b>, and only if all

these episodes were frequent.

Counting the frequency:– slide window over stream

– use smart update technique for the supports

<a> <c>

<a,a> <a,b> <a,c> <b,a> <b,b> <b,c> <c,a> <c,b> <c,c>

<a,a,a> <a,a,b> <a,a,c> <a,b,a> <a,b,b> <a,b,c> …

<a,a,a,a> <a,a,a,b> … … ……

Number of episodes of length k: ek (e is number of events)

An episode of length k has maximally k sub-sequences of length k-1.

We can count supports by sliding a window over the sequence.

Search space

S = < (a,1),(b,2),(c,4),(b,5),(b,6),(a,7),(b,8),(b,9), (c,13), (a,14), (c,17), (c,18) >

w = 4, minsup = 3

C1 = { <a>, , <c> }

a a ab b b b bc c c c

0 1 2

Example

S = < (a,1),(b,2),(c,4),(b,5),(b,6),(a,7),(b,8),(b,9), (c,13), (a,14), (c,17), (c,18) >

w = 4, minsup = 3

C1 = { <a>, , <c> }

Slide window of length 4 over S:

4-supports: <a>:12, :12, <c>:14


0 1 2

Example

S = < (a,1),(b,2),(c,4),(b,5),(b,6),(a,7),(b,8),(b,9), (c,13), (a,14), (c,17), (c,18) >

w = 4, minsup = 3

C1 = { <a>, , <c> }


4-supports: <a>:12, :12, <c>:14 C2 = { <a,a>, <a,b>, <a,c>, <b,a>, <b,b>, <b,c>, <c,a>,

<c,b>, <c,c> }


0 1 2

Example

S = < (a,1),(b,2),(c,4),(b,5),(b,6),(a,7),(b,8),(b,9), (c,13), (a,14), (c,17), (c,18) >

w = 4, minsup = 3

C1 = { <a>, , <c> }



<c,b>, <c,c> }

4-supports: <a,a>:0 <a,b>:6 <a,c>:2 <b,a>:3

<b,b>:7 <b,c>:3 <c,a>:3 <c,b>:1 <c,c>:3


0 1 2

Example

S = < (a,1),(b,2),(c,4),(b,5),(b,6),(a,7),(b,8),(b,9), (c,13), (a,14), (c,17), (c,18) >

w = 4, minsup = 3

C1 = { <a>, , <c> }



<c,b>, <c,c> }

4-supports: <a,a>:0 <a,b>:6 <a,c>:2 <b,a>:3

<b,b>:7 <b,c>:3 <c,a>:3 <c,b>:1 <c,c>:3 C3 = { <a,b,b>,<b,a,b>,<b,b,a>,<b,b,b>,<b,b,c>,<b,c,a>,

<b,c,c>, <c,c,a>, <c,c,c>}

4-supports: <a,b,b>:2, <b,a,b>:2, <b,b,a>:2, <b,b,b>:2,

<b,b,c>:0, <b,c,a>:0, <b,c,c>:0, <c,c,a>:0, <c,c,c>:0


0 1 2

Example

MinEpi

Very similar algorithm based on other support measure

– minimal occurrence of sequence: smallest window in which the sequence occurs

– support of E = number of minimal occurrences of E with a width less than w

S = < a b c b b a b b c a c c c b b> window length = 5

5-support of < a b b > :

mo-support of < a b b > :

MinEpi





5-support of < a b b > : 5

a b c b b a b b c a c c c b b

mo-support of < a b b >

MinEpi





5-support of < a b b > : 5


mo-support of < a b b > : 2

Sequential Pattern Mining: Summary

Mining sequential episodes Two definitions of support:

– w-support

– mo-support

Two algorithms:– WinEpi

– MinEpi

Based on monotonicity principle– generate candidates levelwise

– only count candidates without infrequent subsequences

Outline






– process mining

Other types of patterns

Sequence problems– Strings

– Other types of sequences

– Oher patterns in sequences

Graphs– Molecules

– WWW

– Social Networks

…

Other Types of Sequences

CGATGGGCCAGTCGATACGTCGATGCCGATGTCACGA

Other Patterns in Sequences

Substrings Regular expressions (bb|[^b]{2}) Partial orders Directed Acyclic Graphs

Graphs

Patterns in Graphs

Rules

f: 5 f: 8

f: 4

f: 7

f: 4

0.8 0.5

f: 4

0.57

Summary

What is data mining and why is it important.– huge volumes of data– not enough human analysts

Pattern discovery as an important descriptive data mining task– association rule mining– sequential pattern mining

Important principles:– Apriori principle– breadth-first vs depth-first algorithms

Many kinds and variaties of data-types, pattern types,support measures, …

Data Mining Toon Calders

Documents

Transcript of Data Mining Toon Calders