Post on 28-Oct-2019
The validation processLiterature review
Methodological proposals
The validation of Credit Rating and ScoringModels
Raffaella Calabrese
raffaella.calabrese1@unimib.itUniversity of Milano-Bicocca
Swiss Statistics MeetingGeneva, SwitzerlandOctober 29th, 2009
Raffaella Calabrese Validation of internal rating systems
The validation processLiterature review
Methodological proposals
Outline
1 The validation process
2 Literature reviewCumulative Accuracy Profile CurveReceiver Operating Characteristic Curve
3 Methodological proposalsCurve of Classification Error Costs and Error Costs
Raffaella Calabrese Validation of internal rating systems
The validation processLiterature review
Methodological proposals
Outline
1 The validation process
2 Literature reviewCumulative Accuracy Profile CurveReceiver Operating Characteristic Curve
3 Methodological proposalsCurve of Classification Error Costs and Error Costs
Raffaella Calabrese Validation of internal rating systems
The validation processLiterature review
Methodological proposals
Outline
1 The validation process
2 Literature reviewCumulative Accuracy Profile CurveReceiver Operating Characteristic Curve
3 Methodological proposalsCurve of Classification Error Costs and Error Costs
Raffaella Calabrese Validation of internal rating systems
The validation processLiterature review
Methodological proposals
Preliminary definitions
Credit rating and scoring models estimate the creditobligor’s worthiness and provide an assessment of theobligor’s future status.
The discriminatory power of a rating or scoring modeldenotes its ability to discriminate ex ante betweendefaulting and non-defaulting borrowers.
The validation process assesses the discriminatory powerof a rating or scoring model
Raffaella Calabrese Validation of internal rating systems
The validation processLiterature review
Methodological proposals
Preliminary definitions
Credit rating and scoring models estimate the creditobligor’s worthiness and provide an assessment of theobligor’s future status.
The discriminatory power of a rating or scoring modeldenotes its ability to discriminate ex ante betweendefaulting and non-defaulting borrowers.
The validation process assesses the discriminatory powerof a rating or scoring model
Raffaella Calabrese Validation of internal rating systems
The validation processLiterature review
Methodological proposals
Preliminary definitions
Credit rating and scoring models estimate the creditobligor’s worthiness and provide an assessment of theobligor’s future status.
The discriminatory power of a rating or scoring modeldenotes its ability to discriminate ex ante betweendefaulting and non-defaulting borrowers.
The validation process assesses the discriminatory powerof a rating or scoring model
Raffaella Calabrese Validation of internal rating systems
The validation processLiterature review
Methodological proposals
Cumulative Accuracy Profile CurveReceiver Operating Characteristic Curve
Each borrower is characterized by two random variables:
the score S assigned to the borrower is a continuous r. v.with support (−∞,∞)
the Bernoulli r.v. B represents the borrower’s state at theend of a fixed time-period
B =
{
1, the borrower’s state is default (d );0, the borrower’s state is non default (n).
The conditional distribution functions of S given a value of Bare denoted respectively by Fd(·) and Fn(·).
Raffaella Calabrese Validation of internal rating systems
The validation processLiterature review
Methodological proposals
Cumulative Accuracy Profile CurveReceiver Operating Characteristic Curve
Each borrower is characterized by two random variables:
the score S assigned to the borrower is a continuous r. v.with support (−∞,∞)
the Bernoulli r.v. B represents the borrower’s state at theend of a fixed time-period
B =
{
1, the borrower’s state is default (d );0, the borrower’s state is non default (n).
The conditional distribution functions of S given a value of Bare denoted respectively by Fd(·) and Fn(·).
Raffaella Calabrese Validation of internal rating systems
The validation processLiterature review
Methodological proposals
Cumulative Accuracy Profile CurveReceiver Operating Characteristic Curve
Each borrower is characterized by two random variables:
the score S assigned to the borrower is a continuous r. v.with support (−∞,∞)
the Bernoulli r.v. B represents the borrower’s state at theend of a fixed time-period
B =
{
1, the borrower’s state is default (d );0, the borrower’s state is non default (n).
The conditional distribution functions of S given a value of Bare denoted respectively by Fd(·) and Fn(·).
Raffaella Calabrese Validation of internal rating systems
The validation processLiterature review
Methodological proposals
Cumulative Accuracy Profile CurveReceiver Operating Characteristic Curve
The distribution function of the score S is
F (s) = pFd(s) + (1 − p)Fn(s)
where p is the probability of default p = P[B = d ].
The accuracy (AC) is
AC = pFd(s) + (1 − p)[1 − Fn(s)] = 2pFd(s) − F (s) + (1 − p)
Raffaella Calabrese Validation of internal rating systems
The validation processLiterature review
Methodological proposals
Cumulative Accuracy Profile CurveReceiver Operating Characteristic Curve
The distribution function of the score S is
F (s) = pFd(s) + (1 − p)Fn(s)
where p is the probability of default p = P[B = d ].
The accuracy (AC) is
AC = pFd(s) + (1 − p)[1 − Fn(s)] = 2pFd(s) − F (s) + (1 − p)
Raffaella Calabrese Validation of internal rating systems
The validation processLiterature review
Methodological proposals
Cumulative Accuracy Profile CurveReceiver Operating Characteristic Curve
Actual default Actual non defaultPredicted default True Default (TD) False Default (FD)(below s) Type II errorPredicted non default False Non Default (FN) True Non Default(above s) Type I error (TN)
Nd Nn
hit rate F̂d(s) =TDNd
false alarm rate F̂n(s) =FDNn
F̂ (s) = p̂F̂d(s) + (1 − p̂)F̂n(s) =TD + FDNd + Nn
where p̂ =Nd
Nd + Nn
Raffaella Calabrese Validation of internal rating systems
The validation processLiterature review
Methodological proposals
Cumulative Accuracy Profile CurveReceiver Operating Characteristic Curve
Actual default Actual non defaultPredicted default True Default (TD) False Default (FD)(below s) Type II errorPredicted non default False Non Default (FN) True Non Default(above s) Type I error (TN)
Nd Nn
hit rate F̂d(s) =TDNd
false alarm rate F̂n(s) =FDNn
F̂ (s) = p̂F̂d(s) + (1 − p̂)F̂n(s) =TD + FDNd + Nn
where p̂ =Nd
Nd + Nn
Raffaella Calabrese Validation of internal rating systems
The validation processLiterature review
Methodological proposals
Cumulative Accuracy Profile CurveReceiver Operating Characteristic Curve
Actual default Actual non defaultPredicted default True Default (TD) False Default (FD)(below s) Type II errorPredicted non default False Non Default (FN) True Non Default(above s) Type I error (TN)
Nd Nn
hit rate F̂d(s) =TDNd
false alarm rate F̂n(s) =FDNn
F̂ (s) = p̂F̂d(s) + (1 − p̂)F̂n(s) =TD + FDNd + Nn
where p̂ =Nd
Nd + Nn
Raffaella Calabrese Validation of internal rating systems
The validation processLiterature review
Methodological proposals
Cumulative Accuracy Profile CurveReceiver Operating Characteristic Curve
Actual default Actual non defaultPredicted default True Default (TD) False Default (FD)(below s) Type II errorPredicted non default False Non Default (FN) True Non Default(above s) Type I error (TN)
Nd Nn
hit rate F̂d(s) =TDNd
false alarm rate F̂n(s) =FDNn
F̂ (s) = p̂F̂d(s) + (1 − p̂)F̂n(s) =TD + FDNd + Nn
where p̂ =Nd
Nd + Nn
Raffaella Calabrese Validation of internal rating systems
The validation processLiterature review
Methodological proposals
Cumulative Accuracy Profile CurveReceiver Operating Characteristic Curve
Cumulative Accuracy Profile (CAP) Curve andAccuracy Ratio (AR)
curve: CAP(u) = Fd [F−1(u)], u ∈ (0, 1)
Raffaella Calabrese Validation of internal rating systems
The validation processLiterature review
Methodological proposals
Cumulative Accuracy Profile CurveReceiver Operating Characteristic Curve
Cumulative Accuracy Profile (CAP) Curve andAccuracy Ratio (AR)
curve: CAP(u) = Fd [F−1(u)], u ∈ (0, 1)
Raffaella Calabrese Validation of internal rating systems
The validation processLiterature review
Methodological proposals
Cumulative Accuracy Profile CurveReceiver Operating Characteristic Curve
Cumulative Accuracy Profile (CAP) Curve andAccuracy Ratio (AR)
synthetic index (BCBS, 2005):
AR =aR
aR + aQAR ∈ [0, 1]
optimal cut-off score (Hong, 2009): the intersection of theCAP curve and the iso-performance tangent line
Fd(s) =1
2p[F (s) + AC + p − 1]
drawbacks:- dependence on the sample relative frequency of defaulted
borrowers;- the type II error and the costs of wrong classification are
ignored.
Raffaella Calabrese Validation of internal rating systems
The validation processLiterature review
Methodological proposals
Cumulative Accuracy Profile CurveReceiver Operating Characteristic Curve
Cumulative Accuracy Profile (CAP) Curve andAccuracy Ratio (AR)
synthetic index (BCBS, 2005):
AR =aR
aR + aQAR ∈ [0, 1]
optimal cut-off score (Hong, 2009): the intersection of theCAP curve and the iso-performance tangent line
Fd(s) =1
2p[F (s) + AC + p − 1]
drawbacks:- dependence on the sample relative frequency of defaulted
borrowers;- the type II error and the costs of wrong classification are
ignored.
Raffaella Calabrese Validation of internal rating systems
The validation processLiterature review
Methodological proposals
Cumulative Accuracy Profile CurveReceiver Operating Characteristic Curve
Cumulative Accuracy Profile (CAP) Curve andAccuracy Ratio (AR)
synthetic index (BCBS, 2005):
AR =aR
aR + aQAR ∈ [0, 1]
optimal cut-off score (Hong, 2009): the intersection of theCAP curve and the iso-performance tangent line
Fd(s) =1
2p[F (s) + AC + p − 1]
drawbacks:- dependence on the sample relative frequency of defaulted
borrowers;- the type II error and the costs of wrong classification are
ignored.
Raffaella Calabrese Validation of internal rating systems
The validation processLiterature review
Methodological proposals
Cumulative Accuracy Profile CurveReceiver Operating Characteristic Curve
Cumulative Accuracy Profile (CAP) Curve andAccuracy Ratio (AR)
synthetic index (BCBS, 2005):
AR =aR
aR + aQAR ∈ [0, 1]
optimal cut-off score (Hong, 2009): the intersection of theCAP curve and the iso-performance tangent line
Fd(s) =1
2p[F (s) + AC + p − 1]
drawbacks:- dependence on the sample relative frequency of defaulted
borrowers;- the type II error and the costs of wrong classification are
ignored.
Raffaella Calabrese Validation of internal rating systems
The validation processLiterature review
Methodological proposals
Cumulative Accuracy Profile CurveReceiver Operating Characteristic Curve
Receiver Operating Characteristic (ROC) Curve andArea Under the Curve (AUC)
Curve: ROC(u) = Fd [F−1(u)], u ∈ (0, 1)
Raffaella Calabrese Validation of internal rating systems
The validation processLiterature review
Methodological proposals
Cumulative Accuracy Profile CurveReceiver Operating Characteristic Curve
Receiver Operating Characteristic (ROC) Curve andArea Under the Curve (AUC)
Curve: ROC(u) = Fd [F−1(u)], u ∈ (0, 1)
Raffaella Calabrese Validation of internal rating systems
The validation processLiterature review
Methodological proposals
Cumulative Accuracy Profile CurveReceiver Operating Characteristic Curve
Synthetic index (BCBS, 2005):
AUC =
∫ 1
0Fd [Fn(s)]dFn(s) AUC ∈ [0.5, 1]
Optimal cut-off score (Hong, 2009): the intersection of theROC curve and the iso-performance tangent line
Fd(s) =1 − p
pFn(s) +
1p
(AC + p − 1).
drawbacks: the costs of wrong classification are ignored.
Raffaella Calabrese Validation of internal rating systems
The validation processLiterature review
Methodological proposals
Cumulative Accuracy Profile CurveReceiver Operating Characteristic Curve
Synthetic index (BCBS, 2005):
AUC =
∫ 1
0Fd [Fn(s)]dFn(s) AUC ∈ [0.5, 1]
Optimal cut-off score (Hong, 2009): the intersection of theROC curve and the iso-performance tangent line
Fd(s) =1 − p
pFn(s) +
1p
(AC + p − 1).
drawbacks: the costs of wrong classification are ignored.
Raffaella Calabrese Validation of internal rating systems
The validation processLiterature review
Methodological proposals
Cumulative Accuracy Profile CurveReceiver Operating Characteristic Curve
Synthetic index (BCBS, 2005):
AUC =
∫ 1
0Fd [Fn(s)]dFn(s) AUC ∈ [0.5, 1]
Optimal cut-off score (Hong, 2009): the intersection of theROC curve and the iso-performance tangent line
Fd(s) =1 − p
pFn(s) +
1p
(AC + p − 1).
drawbacks: the costs of wrong classification are ignored.
Raffaella Calabrese Validation of internal rating systems
The validation processLiterature review
Methodological proposalsCurve of Classification Error Costs and Error Costs
Curve of Classification Error Costs (CEC) and ErrorCosts (EC)
Curve:
C[u] =CFN
2{1 − Fd [F−1(u)]} +
CFD
2Fn[F−1(u)] u ∈ (0, 1)
Synthetic index:
EC∗ =
∫ 1
0C[F (s)]dF (s)
EC =EC∗ − EC∗
R
EC∗
P − EC∗
REC ∈ [0, 1]
where EC∗
R and EC∗
P are respectively the error costs of therandom and perfect models.
Raffaella Calabrese Validation of internal rating systems
The validation processLiterature review
Methodological proposalsCurve of Classification Error Costs and Error Costs
Curve of Classification Error Costs (CEC) and ErrorCosts (EC)
Curve:
C[u] =CFN
2{1 − Fd [F−1(u)]} +
CFD
2Fn[F−1(u)] u ∈ (0, 1)
Synthetic index:
EC∗ =
∫ 1
0C[F (s)]dF (s)
EC =EC∗ − EC∗
R
EC∗
P − EC∗
REC ∈ [0, 1]
where EC∗
R and EC∗
P are respectively the error costs of therandom and perfect models.
Raffaella Calabrese Validation of internal rating systems
The validation processLiterature review
Methodological proposalsCurve of Classification Error Costs and Error Costs
Curve of Classification Error Costs (CEC) and ErrorCosts (EC)
Optimal cut-off score: the value s that satisfies
mins
{
CFN
2[1 − Fd(s)] +
CFD
2Fn(s)
}
= maxs
[
Fd(s)
CFD−
Fn(s)
CFN
]
Point measure (Zenga, 2007): U(c) =
−
µ (c)+µ (c)
where c ∈ (−∞,+∞) and
−
µ (c) =1
F (c)
∫ c
−∞
{
CFN
2[1 − Fd(s)] +
CFD
2Fn(s)
}
dF (s)
+µ (c) =
11 − F (c)
∫
+∞
c
{
CFN
2[1 − Fd(s)] +
CFD
2Fn(s)
}
dF (s)
Raffaella Calabrese Validation of internal rating systems
The validation processLiterature review
Methodological proposalsCurve of Classification Error Costs and Error Costs
Curve of Classification Error Costs (CEC) and ErrorCosts (EC)
Optimal cut-off score: the value s that satisfies
mins
{
CFN
2[1 − Fd(s)] +
CFD
2Fn(s)
}
= maxs
[
Fd(s)
CFD−
Fn(s)
CFN
]
Point measure (Zenga, 2007): U(c) =
−
µ (c)+µ (c)
where c ∈ (−∞,+∞) and
−
µ (c) =1
F (c)
∫ c
−∞
{
CFN
2[1 − Fd(s)] +
CFD
2Fn(s)
}
dF (s)
+µ (c) =
11 − F (c)
∫
+∞
c
{
CFN
2[1 − Fd(s)] +
CFD
2Fn(s)
}
dF (s)
Raffaella Calabrese Validation of internal rating systems
The validation processLiterature review
Methodological proposalsCurve of Classification Error Costs and Error Costs
Bibliography
Basel Committee on Banking Supervision (2005). Studies on theValidation of Internal Rating Systems. Working paper 14. Basel,BIS.
Hong C. S. (2009). Optimal Threshold from ROC and CAPCurves. Communications in Statistics, 38, 2060-2072.
Zenga M. (2007). Inequality curve and inequality index based onthe ratio between lower and upper arithmetic means. Statistica &Applicazioni, V, 3-27.
Raffaella Calabrese Validation of internal rating systems