Post on 19-Dec-2015
22
Outline
Direct Marketing
Review: Evaluation: Lift, Gains
KDD Cup 1997
Lift and Benefit estimation
Privacy and Data Mining
33
Direct Marketing Paradigm
Find most likely prospects to contact
Not everybody needs to be contacted
Number of targets is usually much smaller than number of prospects
Typical Applications retailers, catalogues, direct mail (and e-mail)
customer acquisition, cross-sell, attrition
...
44
Direct Marketing Evaluation
Accuracy on the entire dataset is not the right measure
Approach develop a target model
score all prospects and rank them by decreasing score
select top P% of prospects for action
Evaluate Performance on top P% using Gains and Lift
CPH (Gains): Random List vs Model-ranked
list
0102030405060708090
100
5
15 25 35 45 55 65 75 85 95
RandomModel
5% of random list have 5% of targets,
but 5% of model ranked list have 21% of targets CPH(5%,model)=21%.
Pct list
Cum
ulative %
Hits
Lift Curve
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
5
15 25 35 45 55 65 75 85 95
Lift
Lift(P) = CPH(P) / P
P -- percent of the list
Lift (at 5%)
= 21% / 5%
= 4.2betterthan random
KDD-CUP 1997
Task: given data on past responders to fund-raising, predict most likely responders for new campaign
Population of 750K prospects 10K responded to a broad campaign mailing
(1.4% response rate)
Analysis file included a stratified (non-random) sample of 10K responders and 26K non-responders (28.7% response
rate)
75% used for learning; 25% used for validation
target variable removed from the validation data set
KDD-CUP 1997 Data Set
321 fields/variables with ‘sanitized’ names and labels Demographic information
Credit history
Promotion history
Significant effort on data preprocessing leaker detection and removal
KDD-CUP Participant Statistics
45 companies/institutions participated 23 research prototypes
22 commercial tools
16 contestants turned in their results 9 research prototypes
7 commercial tools
KDD-CUP Algorithm Statistics
Algorithm # of Entries Ave. Score
Rules 2 87
k-NN 1 85
Bayesian 3 83
Multiple/Hybrid 4 79
Other 2 68
Decision Tree 4 44
Of the 16 software/tools… (Score as % of best)
1111
KDD Cup 97 Evaluation
Best Gains at 40% Urban Science
BNB
Mineset
Best Gains at 10% BNB
Urban Science
Mineset
KDD-CUP 1997 Awards
The GOLD MINER GOLD MINER award is jointly shared by two contestants this year
1) Charles ElkanCharles Elkan, Ph.D. from University of California, San , Ph.D. from University of California, San Diego Diego with his software BNB, Boosted Naive Bayesian BNB, Boosted Naive Bayesian ClassifierClassifier
1) Urban Science Applications, IncUrban Science Applications, Inc. . with their software gain, Direct Marketing Selection Systemgain, Direct Marketing Selection System
The BRONZE MINER BRONZE MINER award went to the runner-up
3) Silicon Graphics, IncSilicon Graphics, Inc with their software MineSetMineSet
KDD-CUP Results Discussion
Top finishers very close
Naïve Bayes algorithm was used by 2 of the top 3 contestants (BNB and MineSet)
BNB and MineSet did little data preprocessing
MineSet used a total of 6 variables in their final model
Urban Science implemented a tremendous amount of automated data preprocessing and exploratory data analysis and developed more than 50 models in an automated fashion to get to their results
1414
KDD Cup 1997: Top 3 results
Top 3 finishersare very close
1515
KDD Cup 1997 – worst results
Note that the worstresult (C6) was actuallyworse than random.
Competitor names werekept anonymous,apart from top 3 winners
1616
Better Model Evaluation?
Comparing Gains at 10% and 40% is ad-hoc
Are there more principled methods? Area Under the Curve (AUC) of Gains Chart
Lift Quality
Ultimately, financial measures: Campaign Benefits
1717
Model Evaluation: AUC
Area Under the Curve (AUC) is defined as the
Difference between Gains and Random Curves
Selection
Cum
% H
its
1818
Model Evaluation: Lift Quality
See Measuring Lift Quality in Database Marketing, Piatetsky-Shapiro and Steingold, SIGKDD Explorations, December 2000 .
AUC(Model) – AUC(Random)LQ = ----------------------------- AUC(Perfect) –AUC(Random)
1919
Lift Quality (Lquality)
For a perfect model, Lquality = 100%
For a random model, Lquality = 0
For KDD Cup 97, Lquality(Urban Science) = 43.3%
Lquality(Elkan) = 42.7%
However, small differences in Lquality are not significant
2020
Estimating Profit: Campaign Parameters
Direct Mail example N -- number of prospects, e.g. 750,000
T -- fraction of targets, e.g. 0.014
B -- benefit of hitting a target, e.g. $20 Note: this is simplification – actual benefit will vary
C -- cost of contacting a prospect, e.g. $0.68
P -- percentage selected for contact, e.g. 10%
Lift(P ) -- model lift at P , e.g. 3
2121
Contacting Top P of Model-Sorted List Using previous example, let selection be P = 10% and Lift(P)
= 3
Selection size = N P , e.g. 75,000
Random has N P T targets in first P list, e.g. 1,050
Q: How many targets are in model P-selection?
Model has more by a factor Lift(P) or N P T Lift(P) targets in the selection, e.g. 3,150
Benefit of contacting the selection is N P T Lift(P) B , e.g. $63,000
Cost of contacting N P is N P C , e.g. $51,000
2222
Profit of Contacting Top P
Profit(P) = Benefit(P) – Cost(P) =
N P T Lift(P) B - N P C =
NP (T Lift(P) B - C ) e.g. $12,000
Q: When is Profit Positive?
CLift(P) > ------ , e.g. 2.4 T ·B
When T • Lift(P) B > C , or
Finding Optimal Cutoff
-60
-40
-20
0
20
40
60
10 20 30 40 50 60 70 80 90 100
Est Payoff
Use the formula to estimate benefit for each PFind optimal P
2424
*Feasibility Assessment
Expected Profit(P) depends on known Cost C,
Benefit B,
Target Rate T
and unknown Lift(P)
To compute Lift(P) we need to get all the data, load it, clean it, ask for correct data, build models, ...
2525
*Can Expected Lift be estimated ?
only from N and T ?
In theory -- no, but in many practical applications,
?!?! surprisingly yes ?!?!
2626
*Empirical Observations about Lift
For good models, usually Lift(P) is monotically decreasing with P
Lift at fixed P (e.g. 0.05) is usually higher for lower T
Special point P = T
for a perfect predictor, all targets are in the first T of the list, for a maximum lift of 1/T
What can we expect compared to 1/T ?
2727
*Meta Analysis of Lift
26 attrition & cross-sell problems from finance and telecom domains
N ranges from 1,000 to 150,000
T ranges from 1% to 22%
No clear relation to N, but there is dependence on T
2828
*Results: Lift(T) vs 1/T
Best Model (R2 = 0.86)
log10(Lift(T)) = -0.05 + 0.52 log10(1/T)
Approximately
Lift(T) ~ T -0.5 = sqrt (1/T)
Tried several linear and log-linear fits
2929
*Actual Lift(T) vs sqrt(1/T) for All Problems
0
2
4
6
8
10
12
14
0 5 10 15 20 25
100*T%
Lift
Actual lift(T) Est. lift(T) Error = Actual Lift - sqrt(1/T)
Avg(Error) = -0.08
St. Dev(Error) = 1.0
3030
*GPS Lift(T) Rule of Thumb
For targeted marketing campaigns,
where 0.01 < T < 0.25,
Lift(T) = sqrt (1/T) 1
Exceptions for
truly predictable or random behaviors
poor models
information leakers
3131
*Estimating Entire Curve
Cumulative Percent Hits
CPH(P) = Lift(P) * P
CPH is easier to model than Lift
Several regressions for all CPH curves
Best results with regression
log10(CPH(P)) = a + b log10(P)
Average R2 = 0.97
3232
*CPH Curve Estimate
Approximately
CPH(P) ~ sqrt(P)
bounds:
P 0.6 < CPH(P) < P 0.4
3333
*Lift Curve Estimate
Since Lift(P) = CPH(P)/P
Lift(P) ~ 1/sqrt(P)
bounds:
(1/P ) 0.4 < Lift(P) < (1/P ) 0.6
3434
*More onEstimating Lift and Profitability
G. Piatetsky-Shapiro, B. Masand, Estimating Campaign Benefits and Modeling Lift, Proc. KDD-99, ACM. www.KDnuggets.com/gpspubs/
3535
KDD Cup 1998
Data from Paralyzed Veterans of America (charity)
Goal: select mailing with the highest profit
Winners: Urban Science, SAS, Quadstone see full results and winner’s presentations at
www.kdnuggets.com/meetings/kdd98
3636
KDD-CUP-98 Analysis UniverseParalyzed Veterans of America (PVA), a not-for-profit organization that provides programs and services for US veterans with spinal cord injuries or disease, generously provided the data set PVA’s June 97 fund raising mailing, sent to 3.5 million
donors, was selected as the competition data
Within this universe, a group of 200K “Lapsed” donors was of particular interest to PVA. “Lapsed” donors are individuals who made their last donation to PVA 13 to 24 months prior to the mailing
3737
KDD Cup-98 Example
Evaluation: Expected profit maximization with a mailing cost of $0.68
Sum of (actual donation-$0.68) for all records with predicted/ expected donation > $0.68
Participant with the highest actual sum wins
3838
KDD Cup Cost Matrix
Predicted Donation
Yes No
Actual
Donation
Yes DonationAmt-0.68
0
No -0.68 0
3939
KDD Cup 1998 Results
Model Selected
Result Rank
GainSmarts
56,330 $14,712
1
SAS 55,838 $14,662
2
Quadstone
57,836 $13,954
3
… … … …
*ALL* 96,367 $10,560
13
… … … …
#20 42,270 $1,706 20
#21 1,551 $ -54 21
Selected: how manywere selected by themodel
Result: the total profit(donations-cost)of the model
*ALL* - selecting all
4040
Summary
KDD Cup 1997 case study
Model Evaluation: AUC and Lift Quality
Estimating Campaign Profit
*Feasibility Assessment GPS Rule of Thumb for Typical Lift Curve
KDD Cup 1998