Ron S. Kenett
description
Transcript of Ron S. Kenett
Relative Linkage Disequilibrium: An intersection between evolution, algebraic statistics, text mining and
contingency tables
Ron S. Kenett KPA Ltd., Raanana, Israel and Department of Statistics and Applied
Mathematics "Diego de Castro", University of Torino, Italy
in collaboration with
Silvia SaliniDepartment of Economics, Business and Statistics, University of Milan, Italy
Text Mining
Outline
Contingency TablesG
raph
ical
Dis
play
s
“Algebraic Statistics”Evolution
Background 1930 - 1975
FISHER, R. A. 1930, The Genetical Theory of Natural Selection. Clarendon, Oxford, U.K..LEWONTIN, R., and KOJIMA, K., 1960, The evolutionary dynamics of complex polymorphisms. Evolution 14: 458-472.KARLIN, S . and FELDMAN, M., 1970, Linkage and selection: Two locus symmetric viability model. Theoretical Population Biology 1 : 39-71.KARLIN, S., 1975, General two-locus models: some objectives, results and interpretations. Theoretical Population Biology 7 : 364-398.KARLIN,S . and KENETT, R. 1977, Variable Spatial Selection with Two Stages of Migration and Comparisons Between Different Timings, Theoretical Population Biology, 11, pp. 386-409.
Linkage Disequilibrium
Evolution
R.A. Fisher Sam KarlinThe Fundamental Theorem of Natural Selection:“the rate of increase of fitness of any organism at any time is equal to its genetic variance at that time”
The Fundamental Theorem of Natural Selection:“the rate of increase of fitness of any organism at any time is equal to its genetic variance at that time”
Background - 1975Contingency Tables
Discrete multivariate analysis: theory and practice“The first comprehensive treatise on the analysis of categorical data using loglinear and related statistical models…”
Discrete multivariate analysis: theory and practice“The first comprehensive treatise on the analysis of categorical data using loglinear and related statistical models…”
Background - 1983Graphical Display
Background - 2008Algebraic Statistics
254 itemsets:The “e” relationship
……….. female…………..infection…………………
……….. female……………………..…………………
……….. ……….…………..infection…………………
……………..………………………….……..…………
57
40
109
48
Text Mining
The DataContingency Tables
e RHS ^RHS
LHS x1=.224 x2=.157
^LHS x3=.429 x4=.189
d RHS ^RHS
LHS x1=.389 x2=.056
^LHS x3=.50 x4=.056
h RHS ^RHS
LHS x1=.057 x2=.321
^LHS x3=.189 x4=.434
LHS
^LHS
RHS^RHS
RHS^RHS
e RHS ^RHS
LHS 57 40
^LHS 109 48
“…a man who has carefully investigated a printed table, finds, when done, that he has only a very faint and partial idea of what he has read; and that like a figure imprinted on sand, is soon totally erased and defaced.”
William Playfair (1786), The Commercial and Political Atlas, from Edward R. Tufte (1983), The Visual Display of Quantitative Information.
Contingency Tables
The SimplexGraphical Display
x1
x2
x3
x4
4
1
1, 0 , 1...4.i ii
x x i
The SimplexGraphical Display
4
1
1, 0 , 1...4.i ii
x x i
Linkage Disequilibrium“Algebraic Statistics”
1x fg D
2 (1 )x f g D
3 (1 )x f g D
4 (1 )(1 )x f g D
1 3f x x
4 2 3D x x x x
RHS ^RHS
LHS x1 x2
^LHS x3 x4
4
1
1, 0 , 1...4.i ii
x x i
1 2g x x
D can be extended to more dimensions…
Two loci, two alleles each, four genotypes:AB, Ab, aB, ab
Two loci, two alleles each, four genotypes:AB, Ab, aB, ab
Linkage Disequilibrium“Algebraic Statistics”
1 2 3 4( , , , )
( ,1 )
( ,1 )
(1, 1)
where
X x x x x
f f f
g g g
e
4
1
1, 0 , 1...4.i ii
x x i
1 3
1 2
f x x
g x x
X f g De e
An algebraic observation…
Relative Linkage Disequilibrium
“Algebraic Statistics”
3 2
3
2
0If Dthenif x x
Dthen RLD
D xD
else RLDD x
1 4
1
4
elseif x x
Dthen RLD
D xD
else RLDD x
M
DRLD
D
DM is the distance from the point corresponding to the contingency table on the surface D=0 in the ee direction, to the surface of the simplex, in that direction.
D is the distance from the point corresponding to the contingency table in the simplex, to the surface D=0 in the ee direction.
Graphical Display
0D M
DRLD
D
MDD
RLD Example
RLD
Contingency Tables
4
2 3
x x
x x
h RHS ^RHS
LHS x1=.057 x2=.321
^LHS x3=.189 x4=.434
e RHS ^RHS
LHS x1=.224 x2=.157
^LHS x3=.429 x4=.189
d RHS ^RHS
LHS x1=.389 x2=.056
^LHS x3=.50 x4=.056
Association Rules
• Association rules are one of the most popular unsupervised data mining methods used in applications such as Market Basket Analysis, to measure the associations between products purchased by each consumer, or in web clickstream analysis, to measure the association between the pages seen (sequentially) by a visitor of a site.
• Mining frequent itemsets and association rules is a popular and well researched method for discovering interesting relations between variables in large databases. The structure of the data to be analyzed is typically referred to as transactional.
• Once obtained, the list of association rules extractable from a given dataset is compared in order to evaluate their importance level. The measures commonly used to assess the strength of an association rule are the indexes of support, confidence, and lift.
Text Mining
Support
• The support for a rule A => B is obtained dividing the number of transactions which satisfy the rule, NA=>B, by the total number of transactions, N.
support {A=>B} = NA=>B / N
Text Mining
SupportText Mining
RHS ^RHS
LHS x1 x2
g
^LHS x3 x4
1-g
f 1-f 1
support {A=>B} = {NA=>B / N} = x1
The higher the support the stronger the information that both type of events occur together.
Confidence
• The confidence of the rule A => B is obtained by dividing the number of transactions which satisfy the rule, NA=>B , by the number of transactions which contain the body of the rule, A.
confidence {A=>B} = NA=>B / NA
Text Mining
ConfidenceText Mining
RHS ^RHS
LHS x1 x2
g
^LHS x3 x4
1-g
f 1-f 1
confidence {A=>B} = {NA=>B / NA}= x1/g
A high confidence that the LHS event leads to the RHS event implies causation or statistical dependence.
Lift
• The lift of the rule A => B is the deviation of the support of the whole rule from the support expected under independence given the supports of the LHS (A) and the RHS (B).
lift {A=>B} = confidence{A=>B} / support{B}= support{A=>B}/support{A}support{B}
Text Mining
Relative Linkage Disequilibrium and other measures
RLD
Sup = 57/254 = .224
Conf = 57/97 = .588
Sup (RHS) = 166/254 = .654
lift = .588/.654 = .90
Contingency Tables
e RHS ^RHS
LHS 57 40
^LHS 109 48
4
2 3
x x
x x
The groceries example
Association Rules
First 20 rules for groceries data, sorted by Lift
The groceries exampleAssociation Rules
Plot of Relative Disequilibrium versus Lift for the 430 rules of groceries data set
For “Lift” > 2.5,RLD varies between 1%-40%
RLD
Lift
The groceries exampleAssociation Rules
The groceries exampleAssociation Rules
Plot of Relative Disequilibrium versus Support for the 430 rules of groceries data set
RLD shows morevariability than “support”RLD
Support
The groceries exampleAssociation Rules
Plot of Relative Disequilibrium versus Confidence for the 430 rules of groceries data set
For RLD of 20%,“Confidence” varies between 1%-40%
RLD
Confidence
The groceries example
Association Rules
First 20 rules for groceries data, sorted by Lift
The groceries exampleAssociation Rules
Plot of Relative Disequilibrium versus Lift for the 430 rules of groceries data set
For “Lift” > 2.5,RLD varies between 1%-40%
RLD
Lift
The groceries exampleAssociation Rules
Plot of RLD versus Chisquare for the top 20 rules of groceries data set, sorted by RLD
RL
D
ChiSquared
The groceries exampleAssociation Rules
Plot of RLD versus Odds Ratio for the top 20 rules of groceries data set, sorted by RLD
RL
D
Odds Ratio
Summary
• RLD is intuitive• RLD yields different answers from
“usual” measures• RLD can be extended to higher
dimensions• There are opportunities in considering
the relationship between Association Rules and Contingency Tables
Text MiningContingency TablesG
raph
ical
Dis
play
s
Algebraic StatisticsEvolution
References• Bishop, Y.M. M., Fienberg, S.E. and Holland, P.W. (1975). Discrete Multivariate Analysis:
Theory and Practice. M.I.T. Press, Cambridge, MA. Paperback edition (1977). Reprinted, by Springer-Verlag, New York (2007).
• Fisher, R. A. (1930). The Genetical Theory of Natural Selection. Clarendon, Oxford, U.K..• Hahsler, M., Grun, B., and Hornik, K. (2005). arules – A computational environment for mining
association rules and frequent item sets. Journal of Statistical Software, 14(15):1–25. ISSN 1548-7660. URL http://www.jstatsoft.org/v14/i15/.
• Karlin, S. and Feldman, M. (1970). Linkage and selection: Two locus symmetric viability model. Theoretical Population Biology 1 : 39-71.
• Karlin, S. and Kenett, R.S. (1977). Variable Spatial Selection with Two Stages of Migration and Comparisons Between Different Timings, Theoretical Population Biology, 11, pp. 386-409.
• Kenett, R. (1983). On an Exploratory Analysis of Contingency Tables. The Statistician, 32, pp. 395-403.
• Kenett, R. and Salini, S. (2008). Relative Linkage Disequilibrium: A New measure for association rules UNIMI - Research Papers in Economics, Business, and Statistics http://services.bepress.com/unimi/statistics/art32/
• Lewontin, R,. C., and Kojima, K. (1960). The evolutionary dynamics of complex polymorphisms. Evolution 14: 458-472.
• Omiecinski, E. (2003). Alternative interest measures for mining associations in databases. IEEE Transactions on Knowledge and Data Engineering, 15(1):57–69.
• Piatetsky-Shapiro, G. (1991). Discovery, analysis, and presentation of strong rules. In: Knowledge Discovery in Databases, pages 229–248.
• Shimada, K., Hirasawa K, and Hu J. (2006) Association Rule Mining with Chi-Squared Test Using Alternate Genetic Network Programming, ICDM2006.
• Tan, P-N, Kumar, V., and Srivastava, J. (2004). Selecting the right objective measure for association analysis. Information Systems, 29(4):293–313.
Backup slides
The telecom systems exampleAssociation Rules
PBX No Severity
Customer Type - English EC2 EC1 ALARM1 ALARM2 ALARM3
90009 2 High Tech SEC08 Security NO_ALARM NP NP
90009 2 High Tech NTC09 Network Communications NO_ALARM NP NP
90009 2 High Tech SEC08 Security NO_ALARM NP NP
90009 2 High Tech SEC08 Security NO_ALARM NP NP
90021 2 Municipalities SEC08 Security NO_ALARM NP NP
90033 2 Transportation SFW05 Software PCM TIME SLOT NP NP
90033 3 Transportation INT04 Interface PCM TIME SLOT NP NP
90033 3 Transportation SEC05 Security PCM TIME SLOT NP NP
90038 2 Municipalities SFW05 Software NO_ALARM NP NP
The telecom systems exampleAssociation Rules
Item Frequency Plot (Support>0.1) of telecom data set
The telecom systems exampleAssociation Rules
3D Simplex Representation for 200 rules of telecom data set and for the top 10 rules sorted by RLD
The telecom systems exampleAssociation Rules
2D Simplex Representation for
the top 10 rules sorted by RLD of telecom data set
The telecom systems exampleAssociation Rules
Top 10 rules sorted by RLD of telecom data set
RLD Statistical Properties
Contingency Tables
RLD Statistical PropertiesContingency Tables