CANE 2007 Spring Meeting Visualizing Predictive Modeling Results Chuck Boucek (312) 879-3859.
-
Upload
lorena-sullivan -
Category
Documents
-
view
215 -
download
0
Transcript of CANE 2007 Spring Meeting Visualizing Predictive Modeling Results Chuck Boucek (312) 879-3859.
-
CANE 2007 Spring MeetingVisualizing Predictive Modeling ResultsChuck Boucek (312) 879-3859
-
AgendaData ValidationHypothesis BuildingModel BuildingModel TestingMonitoringVisualization as a Diagnostic Tool
-
Data ValidationGoalsValidate reasonableness of dataUnderstand key patterns in dataUnderstand changes in data and underlying business through time
-
Data ValidationHistogram is a simple tool to for reasonability testing of modeling database
-
Data ValidationMosaic Plot shows the distribution of predictors in two dimensions
-
Data ValidationMissing Data plot shows the relationship of missing data elements
-
Data ValidationTime series plots identify consistency of data over timeClaims Match to Exposure1993199419951996199719981999200020012002200320040.00.10.20.30.40.50.60.70.80.91.0Company 1Company 2Company 3
-
Hypothesis BuildingGoalsPerform initial analysis of potential predictor variablesLimit the list of predictor variables to be employed in subsequent phases of model buildingFurther reasonability testing of data
-
Frequency0.00.3750.7501.1251.50000.010.020.3Severity100001750025000325004000000.010.020.3Pure Premium5000750010000125001500000.0010.010.020.3Loss Ratio0.2500.3750.5000.6250.75000.010.020.3025005000750010000Exposure ($MM)00.010.020.3050100150200Premium ($MM)00.010.020.3Demographic Variable 10.0010.0010.0010.0010.001
-
Hypothesis BuildingQuantile-Quantile plots help identify needed transformations of data
-
Hypothesis BuildingCorrelation Web concisely summarizes a correlation matrix
-
Model BuildingModel building is an iterative processUnderstanding patterns and relationships throughout this process is critical
-
Model BuildingPartial Plots are a key tool to visualize predictor variables throughout the model building processWhat is a Partial Plot? Linear Predictor = k + b1X1 + b2X2 + b3X3 + b4X4 Predicted value = (ek) x (eb1X1) x (eb2X2) x (eb3X3) x (eb4X4)Partial Plot demonstrates an individual predictor variables contribution to final prediction
-
Model BuildingPartial Plot demonstrates an individual predictor variables contribution to final prediction
-
Model BuildingPartial Plot with modified scatter plot of variable
-
Model BuildingTime Consistency plot is a critical tool for numeric predictors
-
Model BuildingPartial Plot for a factor variable
-
Frequency0.00.3750.7501.1251.500Severity1000017500250003250040000NoYesPure Premium50007500100001250015000Loss Ratio0.2500.3750.5000.6250.750025005000750010000Exposure (Pred. Count)050100150200Premium ($MM)Credit Variable 1NoYesNoYesNoYesNoYesNoYes
-
Model TestingLikely the most critical visualizations in predictive modeling workManagements perception of a projects success will likely depend on these visualizationsHoldout testsCross validation tests
-
Model TestingLift Chart shows overall model performance0.40.50.60.70.80.91.00.40.50.60.70.80.71.0PredictedActualLoss Ratio Lift Chart - Holdout Sample
-
Model TestingROC Curve shows overall model performanceNull, 0Perfect, 1prem, 0.51pred.loss, 0.56Holdout Sample ROC Curve
-
Model TestingClassical Cross Validation exhibit
-
Monitoring Model ResultsThe work does not end when the lift chart looks goodMonitoring toolsDecile managementException analysisModel vs. Actual Results
-
Monitoring Model ResultsDecile ManagementRetentionLoss RatioRate ActionTier/Schedule Mod
-
Monitoring Model ResultsAverage score over time
-
Monitoring Model ResultsLoss ratio of model exceptions
-
Visualization as Diagnostic ToolFrequency and severity models have been developedModel is underperforming in predicting loss ratioLikely cause of underperformance is severity model
-
Visualization as Diagnostic Tool
-
Visualization as Diagnostic Tool
-
Visualization as Diagnostic Tool
-
Visualization as Diagnostic ToolTwo different visualizations of the same model tell a very different story!