sherlock - DA QCRIda.qcri.org/ntang/pubs/sherlock.slides.pdf · Sherlock Rules 7 name dep t nation...

Proof Positive and Negative in Data Cleaning

Matteo InterlandiNan Tang

Sherlock Rules

•Motivation

•Sherlock Rules

•Fundamental problems

•Algorithms

Outline

2

Roadblocks to Get Value from Data?

3

Data Mining

Machine Learning

Rule Discovery

Roadblocks to Get Value from Data?

3

High Quality Data

Data Mining

Machine Learning

Rule Discovery

name nation capitalSi China Beijing

Yan China ShanghaiIan China Tokyo

D



D

consistent D’nation -> capital


Yan China BeijingIan China Beijing

data repairing



D






annotated D”

data repairingproof positive and negative



D






annotated D”


help



D






annotated D”


helpSherlock Rules

•Motivation

•Sherlock Rules


•Algorithms

Outline

5

Proof Positive and Negative

6

name dept

nation capital bornat officePhnSi DA China Beijing ChenYang 28098001

Yan DA China Shanghai Chengdu 24038698Ian ALT China Beijing Hangzhou 33668323

t1t2t3

name officePhn mobileSi 28098001 66700541

Yan 24038698 66706563Ian 27364928 33668323

r1r2r3


6

name dept



t1t2t3


Yan 24038698 66706563Ian 27364928 33668323

r1r2r3

Proof Positive/Negative, Correction

t3[Ian] is correct, t3[officePhn] = 27364928


6

name dept



t1t2t3


Yan 24038698 66706563Ian 27364928 33668323

r1r2r3



Proof Positive/Negative

t3[Ian] is correct, t3[officePhn] is wrong


6

name dept



t1t2t3

country capitalChina BeijingJapan TokyoChile Santiago

s1s2s3






6

name dept



t1t2t3


s1s2s3



Proof Positive

t1[nation, capital] is correct t3[nation, capital] is correct



Sherlock Rules

7

name dept



t1t2t3


Yan 24038698 66706563Ian 27364928 33668323

r1r2r3


s1s2s3

D

Dm

evidence positive

negative

Point of Innovation

8

Integrity Constraints

There does not exist t1[X1] = t2[X2] but

t1[B1] = t2[B2]

(China, Shanghai)

(China, Beijing)

=

<>

Point of Innovation

8

Integrity Constraints

There does not exist t1[X1] = t2[X2] but

t1[B1] = t2[B2]

(China, Shanghai)

(China, Beijing)

Sherlock Rules

t1[X1] = t2[X2] and t1[B] = t2[B-], then

t1[B] := t2[B+]

(China, Shanghai)

(China, Beijing, Shanghai)

=

<>

Applying Multiple Rules

9

+

Pos(t)

Neg(t)

Free(t)-

+

Sherlock Rules in Action

10

t1 (Si, DA, China, Beijing, ChenYang, 28098001)

t1 (Si+, DA, China, Beijing, ChenYang-, 28098001+)

t1 (Si+, DA, China, Beijing, ShenYang+, 28098001+)

Sherlock Rules in Action

10

t1 (Si, DA, China, Beijing, ChenYang, 28098001)

t1 (Si+, DA, China, Beijing, ChenYang-, 28098001+)

t1 (Si+, DA, China, Beijing, ShenYang+, 28098001+)

Pos(t1)

Transformation Rules

11

•Motivation

•Sherlock Rules


•Algorithms

Outline

12

Fundamental Problems

13

Termination

Determinism

Consistency

Implication

(coNP-complete)

(coNP-complete)

•Motivation

•Sherlock Rules


•Algorithms

Algorithms

14

Algorithms

15

Naive Repairing

chase-based

O(|R|x|Sigma|x|M|)

Algorithms

15

Naive Repairing

chase-based

O(|R|x|Sigma|x|M|)

Fast Repairing

Similarity indicesto reduce |M|

(BK-tree, FastSS, n-gram)

Inverted indexto reduce |Sigma|

(hash map)

O(|R|x|Sigma| x com(S))

Algorithms

15

Naive Repairing

chase-based

O(|R|x|Sigma|x|M|)

Fast Repairing

Similarity indicesto reduce |M|

(BK-tree, FastSS, n-gram)

Inverted indexto reduce |Sigma|

(hash map)

O(|R|x|Sigma| x com(S)) Caching similarity index accessesRule pruning based on dependency

Rule Pruning Example

16

R1 R2

R3

R1:R2:R3:t3(Ian, ALT, Chine, Beijing, Hangzhou, 33668323)


16

R1 R2

R3


iteration 1: {(R1, Yes), (R2, Yes), (R3, No)}


16

R1 R2

R3


iteration 2: {(R1, Yes), (R2, No), (R3, No)}



16

R1 R2

R3





17

Conclusion

• Sherlock rules for accurately annotating and repairing data

• Fundamental problems

• Efficient algorithms

17

Conclusion

• Sherlock rules for accurately annotating and repairing data

• Fundamental problems

• Efficient algorithms

Future Work

• Let SQL drive the Sherlock workhorse

• Extend Sherlock rules to more data such as RDF (knowledge bases)

sherlock - DA QCRIda.qcri.org/ntang/pubs/sherlock.slides.pdf · Sherlock Rules 7 name dep t nation...

Documents

Transcript of sherlock - DA QCRIda.qcri.org/ntang/pubs/sherlock.slides.pdf · Sherlock Rules 7 name dep t nation...