Diagnosis tools to estimate Value-at-Risk – Monte Carlo approach

A. Nabbi1, N. Drummen1

Maastricht University

Abstract

Performance of Value-at-Risk models are measured in a Monte Carlo approach by mean of dynamicconditional quantile test, Kupiec, alternative Z-test and Portmanteau statistics, namely Ljung-Box andLi-McLeod. This paper shows the size distortion of backtesting tools above which leads to correct samplesize.

Keywords: Value-at-Risk, Backtesting, Monte Carlo simulations

1. Introduction

Value-at-Risk (hereinafter VaR) is widely used as the standard quantitative tool for measuring marketrisk in financial institutions. It provides a more sensible measure of risk by stressing on losses in aportfolio in comparison with variance as a measure of risk. Although VaR has its own downsides i.e.uncertainty about the underlying distribution. Estimation methods mostly fall into three models, namelyparametric, semi-parametric and non-parametric. The difference is between assumption of the underlyingdistribution. This paper focuses on parametric conditional VaR that employs information up to a specifictime in short horizon.

To investigate the diagnosis tools for VaR estimation, this paper contains a brief definition of VaR insection(2) following by backtesting statistics, namely Kupiec proportion of failure(POF), alternative Ztest, dynamic conditional quantile(DQ), Ljung-Box Q-statistic(QLB) and Li-Mcleod Q-statistic(QLM)in section(3) in order to check the performance of the model. In section(4) Monte Carlo simulation isapplied to VaR estimation. Finally, results and conclusions are provided in sections (5) and (6).

2. VaR Model

The VaR with coverage δ, denoted by V aRδt , is the quantile such that

Pr(xt < −V aRδt |Ωt−1) = δ (1)

Note that the true conditional distribution is unknown. As a result the computation of V aRδt is chal-lenging. Furthermore the so-called hit processes as defined as follows:

HIT δt = ı(xt < −V aRδt |Ωt−1)− δ (2)

where ı(.) is an indicator function. For diagnosing conditional VaR it is crucial to verify that there is noexplanatory content included in the information set Ωt−1.

3. Backtesting statistics

VaR models are constructed to predict future risks precisely. To verify the usefulness of the estimatedmodel the results have to be consistent and reliable. Thus, model has to be backtested. In general,backtesting is a statistical process to compare actual profits and losses to estimated VaR. In particular,backtesting process examine whether the frequency of exceptions lay inside the selected confident level(Nieppola, 2009).

Email addresses: a.nabbi@student.maastrichtuniversity.nl (A. Nabbi),n.drummen@student.maastrichtuniversity.nl (N. Drummen)

Preprint submitted to Elsevier June 8, 2015

3.1. Dynamic conditional quantile test

DQ test is based on the linear regression model on the shocks and set of explanatory variablesincluding a constant and the lags of the shocks (Engle and Manganneli, 2004). Under the conditionalVaR coverage assumption HIT δt must be uncorrelated with its own lags. It is explicit that the expectedvalue of shocks are zero.

HIT δt = β0 +

p∑k=1

βkHITδt−k + εt (3)

where εt is discrete and i.i.d process. Note that the response variable is taking two values. In order toestimate equation (3) Probit regression model is chosen.

For backtesting, DQ test is used to test whether HIT δt (shock at time t) and HIT δt−p (p-order laggedshocks) are orthogonal.

DQ =β′(∑T

i=1Xt−iX′t−i)β

δ(1− δ)∼ χ2

(p+1) (4)

where X is the explanatory variables matrix. This statistic is asymptotically χ2 distributed with p + 1degree of freedom. Rejection of the null hypothesis means weak performance of VaR model. In otherwords, model is not well-specified.

3.2. Kupiec test

Tests of unconditional coverage e.g. Kupiec test, alternative Z-test, are cornerstone of accuracy mea-surement of the VaR models. Rejection of null hypothesis in Kupiec POF test means that considerablefraction of VaR violations is different than δ in corresponding sample (Campbell, 2005).

Kupiec = 2 log((1− δ

1− δ)T−I(δ)

.( δδ

)I(δ))δ =

1

TI(δ)

I(δ) =

T∑t=0

ı(xt < −V aRδt )

(5)

where this statistic is χ2 distributed with one degree of freedom. if the null hypothesis is exceed thecritical values the model is considered inaccurate.

3.3. Alternative Z-test

This test is an alternative approximation to the Kupiec’s POF test. The Z-statistic is based on thesample average number of violations over a given period of time (Campbell ,2005).

z =

√T (δ − δ)√δ(1− δ)

(6)

where z has standard normal distribution. Furthermore the exact finite sample distribution is known.

3.4. Ljung-Box Q statistic

From the Portmanteau statistics QLB test is picked, to test the null hypothesis that the residuals areindependent up to lag p.

QLB = T (T + 2)

p∑k=1

r2kT − k

(7)

where rk is the sample autocorrelation of order k of the residual. This statistic is much closer to χ2(p)

than following classic Box-Pierce statistic (Arranz ,2005).

QBP = T

p∑k=1

r2k (8)

2

3.5. Li-McLeod Q statistic

Another finite sample distribution of test statistic is proposed by Li and McLeod. Notice that QLBand QLM tests go beyond the usual hypothesis that the first p autocorrelations are zero. One advantageof using QLM test is that it moves much closer to its asymptotic distribution without inflating its variancein comparison with QLB (Arranz ,2005).

QLM =p(p− 1)

2T+QBP =

p(p− 1)

2T+ T

p∑k=1

r2k (9)

This statistic is χ2 distributed with p degree of freedom.

4. Monte Carlo simulation

All test statistics with underlying process can be estimated by Monte Carlo simulation under nullhypothesis. Therefore the Monte Carlo significant levels are accurate for VaR diagnosis.

4.1. Critical values

In order to test performance of the model, asymptotic critical values are needed. Each statistic’scritical values in Monte Carlo simulation are picked from their underlying distribution. Moreover, theestimated parameters of Threshold Power GARCH process are tested under the null of being equal tothe assumed true value from estimation in the next section.

The lag order p = 5 is chosen to specify the binary regression and simulated sample size is selectedfrom T ∈ 750, 1000, 2000, 5000, 10000. VaR models are estimated using δ ∈ 1%, 5% coverage.

4.2. Simulated processes

Data processes are generated based on Threshold Power GARCH(1,1). Formally,

xt = σtzt

σt = ω + α+|xt−1|δı(xt−1 > 0) + α−|xt−1|δı(xt−1 ≤ 0) + βσδt−1(10)

where zt is t-Student distributed with v degree of freedom and has a expected value of zero and unitvariance. Using maximum likelihood method the following parameters are estimated:

ω = 0.046879 α+ = −0.033842 α− = 0.18847 β = 0.89615 δ = 1.0198 v = 6.917

This model is the best fitted model to the chosen economic data (S&P500).

5. Results

Table(5) provides informative results from VaR diagnosis tools with 1% and 5% significance level. Incase of 5% coverage level of VaR for threshold Power GARCH when diagnosing at 5% significance levelDQ is far below its nominal level (rejection frequencies ≈ 0.0001%) and Portmanteau statistics(QLBand QLM) are significant and below their nominal level (rejection frequencies ≈ 0.0434% and 0.0431respectively).

Kupiec and alternative Z-test suffer from stronger size distortions when using asymptotic criticalvalues. Moreover, asymptotic critical values are highly sensitive to sample size. As a result, they maynot be accurate in small samples. This can lead to size distortion, in order to fix this issue size-adjustedpower is recommended. With the same argument alternative Z-test is not a good approximation.

According to table (5), there is a correct sample size for which our theoretical findings are confirmed,meaning that all tests have no size distortion under the null hypothesis.

3

Table 1: Rejection frequency of VaR diagnosis tools with significant level of 5% and 1% with coverage 5% and 1%.

δ = 5% TKupiec Z DQ QLB QLM

0.05 0.01 0.05 0.01 0.05 0.01 0.05 0.01 0.05 0.01

Size

750 0.0420 0.0140 0.0720 0.0200 0.0060 0.0060 0.0440 0.0060 0.0400 0.00401000 0.0651 0.0100 0.0962 0.0120 0.0001 0.0020 0.0476 0.0080 0.0471 0.00602000 0.0896 0.0160 0.1441 0.0320 0.0001 0.0000 0.0456 0.0120 0.0452 0.01205000 0.2220 0.0800 0.3435 0.1600 0.0000 0.0000 0.0370 0.0100 0.0370 0.0100

10000 0.4820 0.0300 0.6500 0.0380 0.0000 0.0000 0.0460 0.0001 0.0460 0.0001

δ = 1% TKupiec Z DQ QLB QLM

0.05 0.01 0.05 0.01 0.05 0.01 0.05 0.01 0.05 0.01

Size

750 0.1204 0.0440 0.3040 0.1320 0.0000 0.0000 0.1040 0.0680 0.1040 0.06801000 0.2080 0.0300 0.2920 0.1600 0.0000 0.0000 0.0810 0.0025 0.0810 0.00252000 0.3600 0.1400 0.5220 0.2650 0.0000 0.0000 0.0810 0.0020 0.0810 0.00205000 0.8000 0.4800 0.8950 0.7100 0.0000 0.0000 0.0700 0.0400 0.0700 0.0400

10000 0.9900 0.9400 1.0000 0.9700 0.0000 0.0000 0.0600 0.0300 0.0600 0.0300

6. Conclusion

Diagnosing tools to measure VaR performance are Kupiec, alternative Z-test, Dynamic conditionalquantile, Ljung-Box and Li-McLeod Q statistics. It is shown that Kupiec and alternative Z-test sufferfrom size distortion which is not eliminated by increasing the sample size. In terms of Monte Carlo (sizescenario), enlarging the sample size will lead to finding a correct sample size and prevent the experimentfrom oversizing.

References

[1] O. Nieppola, Backtesting Value-at-Risk Models, Master’s thesis, Helsinki School of Economics, 2009.

[2] R. F. Engle, S. Manganneli, CAViaR: Conditional Autoregressive Value-at-Risk by Regression Quan-tiles .

[3] S. D. Campbell, A Review of Backtesting and Backtesting Procedures .

[4] M. A. Arranz, Portmanteau Test Statistics in Time Series .

4