Post on 10-Mar-2021
Forecasting Based on Unobserved Variables
2014-5
Niels Strange Hansen
PhD Thesis
DEPARTMENT OF ECONOMICS AND BUSINESS
AARHUS UNIVERSITY � DENMARK
Forecasting Based on Unobserved Variables
2014-5
Niels Strange Hansen
PhD Thesis
DEPARTMENT OF ECONOMICS AND BUSINESS
AARHUS UNIVERSITY � DENMARK
FORECASTING BASED ON UNOBSERVEDVARIABLES
By Niels Strange Hansen
A PhD thesis submitted to
School of Business and Social Sciences, Aarhus University,
in partial fulfilment of the requirements of
the PhD degree in
Economics and Business
March 2014
CREATESCenter for Research in Econometric Analysis of Time Series
Data! data! data! I can’t make bricks
without clay.
- Sherlock Holmes
PREFACE
This dissertation is the tangible outcome of my studies as a PhD student at the Depart-
ment of Economics and Business at Aarhus University and was written in the period
from February 2011 to March 2014. I am grateful to the Department of Economics
and Business as well as the Center for Research in Econometric Analysis of Time
Series (CREATES) funded by the Danish National Research Foundation for providing
both an extraordinary research environment and financial support. Personal travel
grants from Oticon Fonden and Knud Højgaards Fond were highly appreciated.
Several people deserve my gratitude. First, I would like to thank my main advisor
Asger Lunde for his guidance, numerous helpful comments, and, in particular, for his
patience. I am very happy to have worked together with Asger on two of the papers in
this dissertation and I hope that we can continue our collaboration in the years to
come. I would like to thank my co-advisor Niels Haldrup for his support and useful
advice regarding various applications. I am very grateful to both my advisors for
believing in me and encouraging me to apply for a PhD position when I was finishing
my master’s studies in 2010.
In 2013 I had the great pleasure of visiting Allan Timmermann at Rady School
of Management at University of California, San Diego. I would like to thank Allan
for inviting me and to thank the Rady School of Management for the hospitality.
During my visit I worked together with Allan on a joint paper with Asger Lunde
and Russ Wermers from University of Maryland. Working together with these highly
accomplished researchers was very inspiring and I am pleased that our joint paper is
a part of this dissertation.
I would like to thank the faculty at the Department of Economics and Business at
Aarhus University and at CREATES for a very active and excellent research environ-
ment. I am very grateful to my fellow PhD students for countless hours of interesting
conversations, funny activities and more coffee breaks than I can count. In particular,
I would like to thank my office mates Anne, Mikkel, Laurent and Tjörvi for coping with
me - occasionally I talk a little too much, sorry. Like numerous PhD students before
me I am very thankful to Johannes for his help with everything computer related, in
particular LATEX, R, Ox and Raspberry Pi, and for many coffee breaks. Anders is thelocal go-to guy for everything related to mathematics, statistics and econometrics and
I am very thankful for his help. Husted, Anders L. and I commenced our studies at the
i
ii
same day in 2005 and I want to thank the two of them for all the funny experiences
we have shared during more than eight years at the university. Finally, the CREATES
corridor would not have been the same without Jonas, Juan Carlos and Manuel.
I am very thankful for the endless support and understanding my family has
shown me during the last three years. Finally, I am very thankful for all the support
and for the sacrifices made by my girlfriend Signe. You have been an amazing support
throughout this process and I am looking forward to returning the favor.
Niels Strange Hansen
Aarhus, March 2014
UPDATED PREFACE
The predefence took place on April 30, 2014. The assessment committee consists
of Jesper Rangvid, Copenhagen Business School, Bradley Steele Paye, University of
Gerogia and Thomas Quistgaard Pedersen, Aarhus University. I am thankful to the
members of the committee for their careful reading of my dissertation and for their
constructive comments and suggestions. Some of the suggestions are incorporated
in the present version of the dissertation while others remain for future research.
Niels Strange Hansen
Aarhus, July 2014
iii
CONTENTS
Summary vii
Dansk resumé ix
1 Analyzing Oil Futures with a Dynamic Nelson-Siegel Model 11.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.2 Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.3 Modelling the Term Structure . . . . . . . . . . . . . . . . . . . . . . . 7
1.4 Empirical Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
1.5 Forecasting the Term Structure . . . . . . . . . . . . . . . . . . . . . . 16
1.6 VaR for Portfolios of Factors . . . . . . . . . . . . . . . . . . . . . . . . 21
1.7 Concluding Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
1.8 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
2 Time-Varying Skills 292.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
2.2 Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
2.3 Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
2.4 Empirical Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
2.5 Forecast Combinations . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
2.6 Model Confidence Set . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
2.7 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
2.8 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
3 Forecasting with Schwartz Models 773.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78
3.2 Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
3.3 Models from Schwartz (1997) . . . . . . . . . . . . . . . . . . . . . . . 80
3.4 Recursive estimation and forecasting . . . . . . . . . . . . . . . . . . . 89
3.5 Empirical Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90
3.6 Concluding Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102
3.7 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102
v
SUMMARY
The first and third chapter in this dissertation consider the modelling and forecasting
of the prices of futures contracts on oil. The second chapter develops a new model for
time-varying mutual fund skills and forecasting of fund performance. Prices of futures
contracts and mutual fund performance presents two different areas of research but
the chapters in this dissertation are linked through their use of unobserved variables
for the purpose of forecasting.
The notion of unobserved variables might seem odd as economists have access to
enormous amounts of data observed in many different markets. It is, however, often
from unobserved variables and their characteristics the most interesting insights are
obtained. This is the case in factor models, see Stock and Watson (2002), where all the
information in a very large dataset is approximated by a few underlying variables, or
factors. Similarly, Diebold and Li (2006) obtain successful forecasts for the term struc-
ture of interest rates by modelling three underlying factors. Schwartz (1997) presents
an excellent example of unobserved variables in commodity markets. In these mar-
kets, spot prices can be very uncertain and sometimes completely unobservable. The
prices of futures or forward contracts, on the other hand, are directly observable
and economic theory tells us about the relationship between the two. The spot price
can then be estimated from the prices of the futures contracts. In the mutual funds
literature, Mamaysky, Spiegel, and Zhang (2007) have successfully introduced a fund
specific unobserved process for manager skill to model fund performance.
The first chapter uses the methodology of Diebold and Li (2006) from the literature
on interest rate modelling to extract three factors from a vast data set consisting of
prices of futures contracts on oil. A highly flexible econometric model is fitted to the
extracted factors and forecasts can be constructed. The forecasts of the factors can
then be used to forecast the prices of futures contracts. A comprehensive real-time
forecasting exercise shows that this approach to forecasting performs significantly
better than conventional benchmarks. We also show that portfolios of oil futures
can be constructed from the extracted factors and that we can successfully calculate
value at risk for such portfolios.
The second chapter develops a unified model of time-varying skills for mutual
funds. In particular, it nests the model of Mamaysky et al. (2007), where skills of
fund managers are considered to be an unobserved variable. When a model for this
vii
viii SUMMARY
variable is specified it can be estimated from the returns of the mutual fund. Forecasts
of skill and in turn forecasts of performance of the fund can then be constructed.
We have access to a unique data set consisting of returns and actual stock holdings
for more than 2.000 mutual funds recorded over time. The model developed in this
chapter allow us to use both returns and holdings-based information to forecast
performance. Forecast combinations are used to show that models which rely on
both returns an holdings-based information performs better than returns-based
models out-of-sample. This chapter also proposes a new methodology to identify the
set of funds with superior performance which is based on the model confidence set
of Hansen, Lunde, and Nason (2011).
The third chapter revisits the classical commodity models of Schwartz (1997).
In this framework prices of futures contracts are observed. It is assumed that the
evolution in the futures prices is determined by one or more unobserved factors. In
particular, the spot price of the commodity and potentially the convenience yield.
By assuming models for the unobserved variables it is possible to derive a model
for the futures prices. Similarly, forecasts of the futures prices can be constructed
from forecasts of the unobserved variables. The models of Schwartz (1997) have
been extended in several directions, but the original models are still widely used. The
chapter investigates the implications for the performance of the different models
from several simplifying assumptions in Schwartz (1997). The results show that
removing these assumptions leads to better forecasting performance.
References
Diebold, F. X., Li, C., 2006. Forecasting the term structure of government bond yields.
Journal of Econometrics 130, 337–364.
Hansen, P., Lunde, A., Nason, J., 2011. The model confidence set. Econometrica 2 (79),
453–497.
Mamaysky, H., Spiegel, M., Zhang, H., 2007. Improved forecasting of mutual fund
alphas and betas. Review of Finance 11, 359–400.
Schwartz, E. S., 1997. The stochastic behavior of commodity prices: Implications for
valuation and hedging. The Journal of Finance 52 (3), 923–973.
Stock, J., Watson, M., 2002. Forecasting using principal component analysis from a
large number of predictors. Journal of The American Statistical Association 97 (460),
1167–1179.
DANSK RESUMÉ
Det første og tredje kapitel i denne afhandling omhandler modellering og fremskriv-
ning af priser på futureskontrakter på olie. Afhandlingens andet kapitel udvikler en ny
model, som beskriver den tidsvarierende ydeevne for investeringsforeninger og frem-
skriver deres præstationer. Priser på futureskontrakter og investeringsforeningers
ydeevne udgør to forskellige forskningsområder, men kapitlerne i denne afhandling
er stadig forbundet. Fremskrivningerne i alle tre kapitler er nemlig baseret på variable,
som ikke kan observeres.
Det kan virke sært at tale om variable, som ikke kan observeres. Specielt for økono-
mer, som har adgang til enorme mængder af observerbart data fra mange forskellige
markeder. Det er dog ofte fra variable, som ikke umiddelbart kan observeres og fra
deres egenskaber, at vi kan lære de mest interessante ting. Dette er netop tilfældet for
faktormodellerne præsenteret i Stock og Watson (2002). Her kan al informationen
i et meget stort datasæt udtrykkes ved et relativt lavt antal underliggende variable,
også kaldet faktorer. Ligeledes opnår Diebold og Li (2006) succesfulde fremskriv-
ninger for terminsstrukturen på rentemarkedet ved at modellere tre underliggende
faktorer. Schwartz (1997) præsenterer et godt eksempel på uobserverbare variable på
råvaremarkederne. På disse markeder kan spotprisen være meget usikker og kan til
tider slet ikke observeres. Priserne på futureskontrakter kan tilgengæld observeres og
økonomisk teori fortæller os, hvordan spot- og futurespriser er forbundet. Spotprisen
kan således estimeres baseret på futurespriserne. I investeringsforeningslitreraturen
har Mamaysky, Spiegel og Zhang (2007) udviklet en model, hvori ydeevne følger en
uobserverbar process og beskriver investeringsforeningers præstationer.
I kapitel 1 benyttes modellen fra Diebold og Li (2006) til at estimere og udtrække
tre faktorer fra et enormt datasæt bestående af priser på futureskontrakter på olie.
Disse beskrives ved hjælp af en meget fleksibel økonometrisk model og faktorerne
kan herefter fremskrives. Baseret på disse kan man fremskrive priserne på futureskon-
trakter. En omfattende fremskrivningsanalyse i realtid viser at denne fremskrivnings-
metode præsterer markant bedre end konventionelle fremskrivningsmetoder. Vi viser
også, hvordan porteføljer af futureskontrakter kan konstrueres fra de estimerede
faktorer. For disse porteføljer viser vi, hvordan value at risk kan udregnes.
Kapitel 2 udvikler en model, som forener litteraturen for tidsvarierende ydeev-
ne for investeringsforeninger. Herunder modellen fra Mamaysky, Spiegel og Zhang
ix
x DANSK RESUMÉ
(2007). I denne model betragtes investeringsforeningernes ydeevne som en uob-
serverbar variabel. En model for denne variabel kan estimeres baseret på investe-
ringsforeningens afkast. Vi har adgang til et unikt datasæt, bestående at tidsserier af
faktiske porteføljebeholdninger og afkast for mere end 2000 investeringsforeninger.
Modellen vi udvikler gør os i stand til at benytte både afkast of porteføljebehold-
ninger når vi fremskriver investeringsforeningernes præstationer. Kombinationer
af fremskrivningsmodeller benyttes til at vise, at modeller baseret på både afkast of
porteføljebeholdninger er bedre til at fremskrive end modeller baseret på afkast alene.
Kapitlet udvikler også en ny metode til at identificere gruppen af investeringsforenin-
ger med de bedste præstationer. Denne metode er baseret på model confidence set
fra Hansen, Lunde og Nason (2011).
Det tredje kapitel fokuserer på de klassiske modeller for råvarepriser i Schwartz
(1997). I disse modeller består datasættet af priser på futureskontrakter. Det antages,
at udviklingen i futurespriser tildels bestemmes af en eller flere underliggende variab-
le. Disse antages at være spotprisen for råvaren og potentielt det såkaldte convenience
yield. Ved at antage en model for disse variable er det muligt at udlede en model for
futurespriserne. Ligeledes kan man fremskrive futurespriserne ud fra fremskrivninger
af de underliggende variable. Selvom modellerne i Schwartz (1997) er blevet udvidet i
mange forskellige retninger, benyttes de oprindelige versioner ofte. Dette kapitel be-
lyser effekterne fra de mange forsimplende antagelser i Schwartz (1997) på kvaliteten
af modellernes fremskrivninger. Resultaterne viser, at modellerne fremskriver bedre,
hvis man fjerner disse antagelser.
Litteratur
Diebold, F. X., Li, C., 2006. Forecasting the term structure of government bond yields.
Journal of Econometrics 130, 337–364.
Hansen, P., Lunde, A., Nason, J., 2011. The model confidence set. Econometrica 2 (79),
453–497.
Mamaysky, H., Spiegel, M., Zhang, H., 2007. Improved forecasting of mutual fund
alphas and betas. Review of Finance 11, 359–400.
Schwartz, E. S., 1997. The stochastic behavior of commodity prices: Implications for
valuation and hedging. The Journal of Finance 52 (3), 923–973.
Stock, J., Watson, M., 2002. Forecasting using principal component analysis from a
large number of predictors. Journal of The American Statistical Association 97 (460),
1167–1179.
CH
AP
TE
R
1ANALYZING OIL FUTURES WITH A DYNAMIC
NELSON-SIEGEL MODEL
Niels S. Hansen
Aarhus University and CREATES
Asger Lunde
Aarhus University and CREATES
Abstract
In this paper the dynamic Nelson-Siegel model is used to model the term structure of
futures contracts on oil and obtain forecasts of prices of these contracts. Three factors
are extracted and modelled in a very flexible framework. The outcome of this exercise
is a class of models which describes the observed prices of futures contracts well and
performs better than conventional benchmarks in realistic real-time out-of-sample
exercises.
1
2 CHAPTER 1. ANALYZING OIL FUTURES WITH A DYNAMIC NELSON-SIEGEL MODEL
1.1 Introduction
In the aftermath of the financial crisis commodity markets have received a lot of
attention. Caballero, Farhi, and Gourinchas (2008) argue that commodity markets
presented a more reliable and attractive form of investment when the financial crisis
spread throughout the financial markets. The growth in economic activity in Brazil,
Russia, India and China has, according to Geman (2005), contributed to the increased
popularity and to increased prices across commodity markets. Predicting prices of
commodity futures contracts is very interesting from an academic perspective and
very valuable for producers, speculators and risk managers. We analyze a large dataset
consisting of prices and time to maturity of futures contracts on oil. We focus on oil
because it is the most traded commodity in the world. It is used in the production
of many products most importantly for petroleum. Oil also serves as an important
indicator for the overall state of the world economy. In this paper we take on some of
the challenges presented by data from commodity markets. We obtain very valuable
insights about the prices of commodity futures and how to forecast them.
The objective in this paper is to develop a class of models which explains data well
and performs well out of sample. This paper makes several contributions. First, we
show that the dynamic Nelson-Siegel model can be used to extract a set of underlying
factors which describe the prices of futures contracts on oil. Second, we model the
factors using the class of GARCH models with Normal Inverse Gaussian innovations
in a very flexible copula framework. Third, we successfully forecast the prices of
futures contracts on oil based on both a conventional mean squared error criterion
and a directional criterion. Finally, we show how our model can be used to calculate
value at risk for portfolios consisting of the factors.
Several papers consider the modelling of oil futures. The traditional approach
is to specify a model for the underlying spot price and derive a term structure for
futures prices based on a no arbitrage argument, see Schwartz (1997) and Geman
(2005) for examples. In this paper we do not consider spot prices or their relationship
to the prices of futures contracts. We focus exclusively on modelling and forecasting
the term structure of futures contracts.
The data in this market, as well as data from other commodity markets, has a
structure which resembles data from fixed income markets. Therefore models from
the fixed income literature, in particular interest rate models, are often applied to
analyze commodity futures contracts. In this paper we apply a classical interest rate
model to prices of oil futures. The model is the dynamic Nelson-Siegel model, Nelson
and Siegel (1987) and Diebold and Li (2006), and the modelling approach is inspired
by Noureldin (2011). The analysis is based on the assumption that the relationship
between futures prices and time to maturity can be described by three latent factors.
The validity of this assumption will be determined by how well our models perform
out of sample. By focussing the analysis on these factors the dynamic Nelson-Siegel
model allows for a substantial dimension reduction which facilitates the analysis.
1.2. DATA 3
The dynamic Nelson-Siegel model is also used with great success in areas other than
yield curve modelling, see for example Chalamandaris and Tsekrekos (2011) and Guo,
Han, and Zhao (2014) where implied volatility is modelled in this framework. In the
commodity literature West (2011) uses the dynamic Nelson Siegel model to estimate
futures prices for long dated contracts on agricultural products. The methodology is,
however, different from the one we apply. West (2011) relies on different versions of
the Nelson-Siegel model to take seasonality and other features into account. Instead
we estimate the latent factors and obtain multivariate time series. The idea is then
to model and forecast these factors in order to forecast futures prices. We show
that models based on the dynamic Nelson-Siegel model performs well in realistic
real-time exercises as forecasting and value at risk analysis.
Using techniques from the copula framework in Patton (2009) and Patton (2012),
we apply a decomposition which allows us to model three univariate time series and
a dependence structure individually. We may then draw on the vast literature on
modelling of univariate time series. In particular, the very flexible class of Normal
Inverse Gaussian GARCH (NIG-GARCH) models presented in Jensen and Lunde
(2001), is shown to describe the data well and at the same time offer accurate forecasts
of the factors. We consider two different models for the dependence..
We leave out a part of our sample for real-time forecast evaluation. In this period
we forecast the factors and hence the term structure of futures prices using our model
and compare the results to two other models. We show that we forecast more precisely
than our benchmarks in terms of mean squared error. Practitioners may not, however,
be interested in the mean squared error of a model but rather the models ability to
accurately predict the direction of the changes of the futures prices. We carry out
an analysis to investigate this desirable property of the different models. Finally, we
show that portfolios of oil futures can be constructed from the estimated factors and
that we can successfully calculate value at risk for these portfolios in our framework.
The rest of this paper is organized as follows. In section 1.2 we provide a thorough
description of the data set analyzed in this paper. Section 1.3 contains a description
of the model. In section 1.4 we present the results of the in-sample analysis. Section
1.5 contains the results of the out-of-sample forecast analysis. Section 1.6 is devoted
to the calculation and backtesting of value at risk. Finally, some concluding remarks
are presented in section 1.7.
1.2 Data
Data from the markets for commodity futures contracts is very different from many
other financial data sets. The unique features of the data make analyzing and fore-
casting challenging tasks. This section serves to illustrate the features of the data and
highlight potential challenges which we have to overcome.
4 CHAPTER 1. ANALYZING OIL FUTURES WITH A DYNAMIC NELSON-SIEGEL MODEL
The data set we consider consists of daily closing prices of monthly futures con-
tracts on oil from Reuters.1 The contracts are on light sweet crude oil (WTI), more
details can be found on the CME Group homepage.2 Every day a number of futures
with different time to maturity are traded. The maturities are approximately one
month apart. Each contract expires on the third trading day prior to the 25th calendar
day in the month before delivery. The first observations are from June 1st 2000 and
the sample ends on December 31st 2012 meaning that our sample consists of 55.123
observations of prices and maturities.
It is important to understand the nature of the data in order to understand the
problems we face in this paper. In order to grasp the structure and complexities of the
data it is useful to consider an example of actual data. In Table 1.1 we have presented
a small example of what the data looks like. This is only a very brief example, but
there are several important things to notice. First, note that only a limited number
of contracts exists on a given day. Consider for example the first row of the table, on
this day a contract with 495 days to delivery does not exist. Such a contract exists on
the next day, though. Secondly, the data contains a number of holes. On 06.09.2010
it was possible to trade in a contract with 494 days to maturity, but no one did. This
means that we have two kinds of holes in the data set contracts that do not exist
and contracts which exist but are not traded. Both pose problems for a traditional
time series analysis which is not well suited for dealing with situations in which the
number of observations changes from day to day. Assume now that on 06.10.2010 we
are interested in forecasting the price of a contract with 492 days to delivery which is
potentially traded on 06.11.2010. Futures contracts are characterized by their time
to maturity, so a good way of forecasting is to consider a time series of prices of
contracts with the same time to maturity. Such a time series is not observed, but it
could be constructed by interpolation, see Diebold and Li (2006) or Noureldin (2011)
for examples from the interest rate literature. Another possibility is to consider the
time series of prices for this particular contract, 2012M (delivery in June 2012) and fit
an autoregressive model. The drawbacks here are that the previous observations of
prices are based on another time to maturity. Furthermore, we have to decide what
to do about missing values. If the contract was introduced to the market recently this
approach might be infeasible due to the limited number of observations. Note, that
we always know the maturities of the contracts which are potentially traded on the
next day.
The model we apply in this paper is introduced by Diebold and Li (2006). They
apply the model to monthly yields on U.S. Treasuries. We notice, that our data resem-
bles the data from their analysis. Therefore, we apply the same model to model the
prices of futures contracts on oil. Diebold and Li (2006) present the most important
1The data is kindly provided by Oxford-Man Institute of Quantitative Finance and obtained fromThomson Reuters Tick History with help from Kevin Sheppard.
2http://www.cmegroup.com/trading/energy/crude-oil/light-sweet-crude_contract_specifications.html
http://www.cmegroup.com/trading/energy/crude-oil/light-sweet-crude_contract_specifications.htmlhttp://www.cmegroup.com/trading/energy/crude-oil/light-sweet-crude_contract_specifications.html
1.2. DATA 5
Table 1.1. Data example.
Price (time to maturity (trading days))Contract name 2012M 2012Z 2013M 2013Z 2014Z
06.07.2010 . . . 81.50 (496) 82.48 (622) - 85.00 (875) 85.65 (1127) . . .06.08.2010 . . . 81.37 (495) 83.10 (621) 83.70 (746) 84.60 (874) 86.22 (1126) . . .06.09.2010 . . . - 84.70 (620) - 85.95 (873) 87.60 (1125) . . .06.10.2010 . . . 83.70 (493) 85.60 (619) - 86.85 (872) 87.91 (1124) . . .
Example of actual data. A subsample of the prices and corresponding maturities for contractstraded between 06.07.2010 and 06.10.2012 for five different contracts. The first part of the namefor each contract is the year of delivery. The letter in the contract name denotes the month ofdelivery. M is June and Z is December.
stylized facts of the yield curve dynamics and argue that their model in theory should
be able to account for all these. We will argue, that the stylized facts presented by
Diebold and Li (2006) are all present in our current data set. Diebold and Li (2006)
consider the five following characteristics:
(1) The average yield curve is increasing and concave.
(2) The yield curve assumes a variety of shapes through time including upward
sloping, downward sloping, humped and inverted humped.
(3) Yield dynamics are persistent, and spread dynamics are much less persistent.
(4) The short end of the yield curve is more volatile than the long end.
(5) Long rates are more persistent than short rates.
We are going to present our data set and adapt the above characteristics to our
framework and argue that prices of oil futures exhibit the same characteristics. To get
an idea of which contracts we have available, we first present a plot of the observed
maturities over time in Figure 1.1. Each dot in this plot indicates an observed price.
Several things are worth noticing. Generally it seems that we have a lot of observa-
tions in the short end throughout the sample, while observations in the long end
are more scarce. Time to maturity is measured in business days. Up until 2007 we
have observations of contracts with maturities around 1700 days. From 2007 we have
contracts with long time to maturity around 2200 days. It seems that, for long maturi-
ties, trading is concentrated in contracts where maturities are about 12 months apart.
This is represented by the diagonal "lines". Transactions for medium long maturities
are concentrated in contracts where maturity is 6 months apart. This can be seen as
the diagonal lines are closer to each other for maturities between 300 and 500 days.
Diebold and Li (2006) analyses a data set of monthly yields and fix maturities by
linear interpolation. This means that they analyze time series of 17 different contracts.
6 CHAPTER 1. ANALYZING OIL FUTURES WITH A DYNAMIC NELSON-SIEGEL MODEL
We do not use linear interpolation which means that we have different maturities on
different days. To enable a comparison of our data to the stylized facts we pool the
data into 23 groups such that we have all prices of contracts with less than 100 days
to maturity in the first group. Similarly all prices for contracts with 100 to 200 days to
maturity are collected in the next group and so on. This division into groups is only
used to highlight some of the stylized facts of Diebold and Li (2006) and not in the
actual analysis.
● ● ● ● ● ● ● ●● ● ● ● ● ● ● ●● ● ●● ● ● ● ● ● ● ● ● ● ● ●● ●● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ●● ●● ● ● ● ● ● ● ● ● ●● ●● ● ● ● ●● ● ● ● ●● ● ● ●● ● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ●● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●● ●● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ●● ● ● ●● ●● ● ● ● ● ● ● ● ● ●● ●● ● ● ● ●● ● ● ● ●● ● ● ●● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ●● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ● ● ● ● ●● ●● ● ● ● ●● ● ● ● ●● ● ● ●● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ●● ● ● ● ● ● ● ●● ● ● ● ● ● ●●●● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ● ● ● ● ●● ●● ● ● ● ●● ● ● ● ●● ● ● ●● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ●● ● ● ● ● ● ● ●● ● ● ● ● ● ●● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ●● ● ● ● ●● ● ● ●● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ●● ● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ●● ●● ●● ● ● ● ● ● ● ● ● ● ● ●● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ●● ● ●● ● ● ● ● ● ● ●● ● ● ●● ●● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ●● ● ●● ● ●● ● ● ● ● ● ● ● ●● ● ●● ●● ● ● ● ● ● ● ● ● ● ● ● ● ●● ●● ● ●● ● ● ● ● ● ● ● ● ● ● ●● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ●● ● ●● ● ● ● ● ● ●● ● ● ●● ●● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ●● ● ●● ● ●● ● ● ● ● ● ● ● ●● ● ●● ●● ● ● ● ● ● ● ● ● ● ● ● ● ●● ●● ● ●● ● ● ● ● ● ● ● ● ● ● ●● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ●● ● ●● ● ● ● ● ● ● ●● ● ● ●● ●● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ●● ● ●● ● ●● ● ● ● ● ● ● ● ●● ● ●● ●● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ●● ●● ● ● ● ● ● ● ● ● ● ● ●● ●● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ●● ● ● ●● ●● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ●● ● ●● ● ●● ● ● ● ● ● ● ● ●● ● ●● ●● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ●● ●● ● ● ● ● ● ● ● ● ● ● ●● ●● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●●● ● ●● ● ● ● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ● ● ● ●● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ● ●●● ● ● ● ● ●● ● ● ● ● ●● ● ●● ● ●● ● ● ● ● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ●● ●● ● ● ● ● ● ● ●● ● ●● ●● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●●● ● ●● ● ● ● ● ● ● ●● ● ●● ● ● ● ● ●● ● ● ● ● ● ● ●● ● ●● ● ● ●● ● ● ● ●● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ● ●●● ● ● ● ● ●● ● ● ● ●●● ● ●● ● ●● ● ● ● ●● ● ● ●● ● ● ● ●● ● ● ●● ● ● ● ●● ● ● ● ●● ● ●● ●● ● ●● ● ● ● ● ●● ● ●● ●● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●●● ● ●● ● ● ● ● ● ● ●● ● ●● ● ● ● ● ●● ● ●● ● ● ● ● ●● ● ●● ● ●● ● ● ● ● ●● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ● ●●● ● ● ● ● ●● ● ● ● ●●● ● ●● ● ●● ● ● ● ●● ● ● ●● ● ● ● ●● ● ●● ● ● ● ● ●● ● ● ● ●● ● ●● ●● ● ●● ● ● ● ● ●● ● ●● ●● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●●● ● ●● ● ● ● ● ● ● ●● ● ●● ● ● ● ● ●● ● ●● ● ● ● ● ●● ● ●● ● ●● ● ● ● ● ●● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ● ●●● ● ● ● ● ●● ● ● ● ●●● ● ●● ● ●● ● ● ● ●● ● ● ●● ● ● ● ●● ● ●● ● ● ● ● ●● ● ● ● ●● ● ●● ●● ● ●● ● ● ● ● ●● ● ●● ●● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●●● ● ●● ● ● ● ● ● ● ●● ● ●● ● ● ● ● ●● ● ●● ● ● ● ● ●● ● ●● ● ●● ● ● ● ● ●● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ● ● ●●● ● ●● ● ●● ● ● ● ●● ● ● ●● ● ● ● ●● ● ●● ● ● ● ● ●● ● ● ● ●● ● ●● ●● ● ●● ● ●● ● ●● ● ●● ●● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●●● ● ●● ● ● ● ● ● ● ●● ● ●● ● ● ● ● ●● ● ●● ● ●● ● ●● ● ●● ● ●● ● ●● ● ●● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ● ● ●●● ● ●● ● ●● ● ● ● ●● ● ●● ● ● ● ● ●● ● ●● ● ●● ● ●● ● ● ● ●● ● ●● ●● ● ●● ● ●● ● ●● ● ●● ●● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ●●● ● ●● ● ● ● ● ● ● ●● ● ●● ● ● ● ● ●● ● ●● ● ●● ● ●● ● ●● ● ●● ● ●● ● ●● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ● ● ●●● ● ●● ● ●● ● ● ● ●● ● ●● ● ● ● ● ●● ● ●● ● ●● ● ●● ● ● ● ●● ● ●● ●● ● ●● ● ●● ● ●● ● ●● ●● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ●●● ● ●● ● ● ● ● ● ● ●● ● ●● ● ● ● ● ●● ● ●● ● ●● ● ●● ● ●● ● ●● ● ●● ● ●● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ● ● ●●● ● ●● ● ●● ● ● ● ●● ● ●● ● ● ● ● ●● ● ●● ● ●● ● ●● ● ● ● ●● ● ●● ●● ● ●● ● ●● ● ●● ● ●● ●● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ●●● ● ●● ● ● ● ● ● ● ●● ● ●● ● ● ● ● ●● ● ●● ● ●● ● ●● ● ●● ● ●● ● ●● ● ●● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ● ● ●●● ● ●● ● ●● ● ● ● ●● ● ●● ● ● ● ● ●● ● ●● ● ●● ● ●● ● ● ● ●● ● ●● ●● ● ●● ● ● ● ● ●● ● ●● ●●●● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ● ● ●● ● ●● ● ● ● ● ●● ● ●● ● ● ● ● ●● ● ●● ● ●● ● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ● ● ●● ● ●● ● ● ● ● ●● ● ● ● ●● ● ●● ●● ● ●● ● ● ● ● ●● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ●●● ● ● ● ● ●● ● ● ● ●● ● ●● ● ● ● ● ●● ● ●● ● ● ● ● ● ● ● ●● ● ●● ● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ● ● ●● ● ●● ● ● ● ● ●● ● ● ● ●● ● ●● ●● ● ●● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ●●● ● ● ● ● ●● ● ● ● ●● ● ●● ● ● ● ● ●● ● ●● ● ● ● ● ● ● ● ●● ● ●● ● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●●● ● ● ●● ● ● ● ● ● ●●● ● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ● ● ●● ● ●● ● ● ● ● ●● ● ● ● ●● ● ●● ●● ● ●● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ●●● ● ● ● ● ●● ● ● ● ●● ● ●● ● ● ● ● ●● ● ●● ● ● ● ● ● ● ● ●● ● ●● ● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ●●● ● ● ●● ● ● ● ● ● ●●● ● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ● ● ●● ● ●● ● ● ● ● ●● ● ● ● ●● ● ●● ●● ● ●● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ●●● ● ● ● ● ●● ● ● ● ●● ● ●● ● ● ● ● ●● ● ●● ● ● ● ● ● ● ● ●● ● ●● ● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ●●● ● ● ●● ● ● ● ● ● ●●● ● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ● ● ●● ● ●● ● ● ● ● ●● ● ● ● ●● ● ●● ●● ● ●● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ●●● ● ● ● ● ●● ● ● ● ●● ● ●● ● ● ● ● ●● ● ●● ● ● ● ● ● ● ● ●● ● ●● ● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ●●● ● ● ●● ● ● ● ● ● ●●● ● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ● ● ●● ● ●● ● ● ● ● ●● ● ● ● ●● ● ●● ●● ● ●● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ●●● ● ● ● ● ●● ● ● ● ●● ● ●● ● ● ● ● ●● ● ●● ● ● ● ● ● ● ● ●● ● ●● ● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ●●● ● ● ●● ● ● ● ●● ●●● ● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ● ● ●● ● ●● ● ● ● ● ●● ● ● ● ●● ● ●● ●● ● ●● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ●●● ● ● ● ● ●● ● ● ● ●● ● ●● ● ● ● ● ●● ● ●● ● ● ● ● ● ● ● ●● ● ●● ● ●● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ●●● ● ● ●● ● ● ● ●● ●●● ● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ● ● ●● ● ●● ● ● ● ● ● ● ● ● ● ●● ● ●
● ●● ● ●● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ●●● ● ● ● ●● ● ● ● ●● ● ●● ● ● ● ● ●● ● ●● ● ● ● ● ● ● ● ●● ● ●● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ●●● ● ● ●● ● ● ● ●● ●●● ● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ● ● ●● ● ●● ● ● ● ● ● ● ● ● ● ●● ● ●● ●● ● ●● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ●● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ●● ● ●● ● ● ● ● ●● ● ●● ● ● ● ● ● ● ● ●● ● ●● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ●●● ● ● ●● ● ● ● ●● ●●● ● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ● ● ●● ● ●● ● ● ● ● ● ● ● ● ● ●● ● ● ●● ● ● ●● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ●●● ● ● ●● ● ● ● ● ● ●●● ● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●●● ● ● ●● ● ● ● ●● ●●● ● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ●● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ●●● ● ● ●● ● ● ● ● ● ●●● ● ● ● ● ●● ● ● ● ●● ● ●● ● ● ●● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●●● ● ● ●● ● ● ● ●● ●●● ● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ●● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ●●● ● ● ●● ● ● ● ● ● ●●● ● ● ● ●● ● ● ● ●● ●● ● ● ●● ● ● ● ●● ● ● ● ● ● ● ● ● ● ●● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●●● ● ● ●● ● ● ● ●● ●●● ● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ●● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ●●● ● ● ●● ● ● ● ● ● ●●● ● ● ● ● ●● ● ● ● ●● ● ●● ● ● ●● ● ● ● ●●● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ●●● ● ● ●● ● ● ● ●● ●●● ● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ●● ● ● ● ●●● ● ● ● ● ● ● ● ● ● ● ● ●●● ● ● ●● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ●●● ● ● ●● ● ● ● ● ● ●●● ● ● ● ● ●● ● ● ● ●● ● ●● ● ● ●● ● ● ● ●●● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ●●● ● ● ●● ● ● ● ●● ●●● ● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ●● ● ● ● ●●● ● ● ● ● ● ● ● ● ● ● ● ●●● ● ● ●● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ●●● ● ● ●● ● ● ● ● ● ●●● ● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ●● ● ● ● ●●● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●●● ● ● ●● ●● ● ●● ●●● ● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ●● ● ● ● ●●● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ●● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ●●● ● ● ●● ● ● ● ● ● ●●● ● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ●● ● ● ● ●●● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●●● ● ● ●● ●● ● ●● ●●● ● ● ● ● ●● ● ● ● ●● ● ● ●● ● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ●● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ●●● ● ● ●● ● ● ● ● ● ●●● ● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ●● ● ● ● ●●● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ●●● ● ● ●● ●● ● ●● ●●● ● ● ● ● ●● ● ● ● ●● ● ● ●● ● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ●● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ●●● ● ● ●● ● ● ● ● ● ●●● ● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ●● ● ● ● ●●● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ●●● ● ● ●● ●● ● ●● ●●● ● ● ● ● ●● ● ● ● ●● ● ● ●● ● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ●● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ●●● ● ● ●● ● ● ● ● ● ●●● ● ● ● ● ●● ● ● ● ●● ● ●● ● ● ●● ● ● ● ●●● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ●●● ● ● ●● ●● ● ●● ●●● ● ● ● ● ●● ● ● ● ●● ● ● ●● ● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ●● ● ● ● ● ● ● ● ●● ● ●●● ● ● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ●● ● ● ● ●● ● ●● ● ● ●● ● ● ● ●●● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ●● ● ●● ●●● ● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ●● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ●● ● ● ● ●● ● ●● ● ● ●● ● ● ● ●●● ● ● ● ●● ● ● ● ● ●●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ●● ● ●● ●●● ● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ●● ● ● ● ● ●● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ●● ● ● ● ●● ● ●● ● ● ●● ● ● ● ●●● ● ● ● ●● ● ● ● ● ●●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ●● ● ●● ●●● ● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ●● ● ● ● ● ●● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ●● ● ● ● ●● ● ●● ● ● ●● ● ● ● ●●● ● ● ● ●● ● ● ● ● ●●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ●● ● ●● ●●● ● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ●● ● ● ● ● ●● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ●● ● ● ● ●● ● ●● ● ● ●● ● ● ● ●●● ● ● ● ●● ● ● ● ● ●●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ●● ● ●● ●●● ● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ●● ● ● ● ● ●● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ●● ● ● ● ●● ● ●● ● ● ●● ● ● ● ●●● ● ● ● ●● ● ● ● ● ●●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ●● ●●● ● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ●● ● ● ● ● ●● ● ●● ●● ● ● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ●● ● ● ● ●● ● ●● ● ● ●● ● ● ● ●●● ● ● ● ●● ● ● ● ● ●●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ●● ●●● ● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ●● ● ● ● ● ●● ● ●● ●● ● ● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ●● ● ● ● ●● ● ●● ● ● ●● ● ● ● ●●● ● ● ● ●● ● ● ● ● ●●● ● ●● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ●● ●●● ● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ●● ● ● ● ● ● ● ● ●● ● ● ●● ● ● ● ● ●● ● ●● ●● ● ● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ●● ● ● ● ●● ● ●● ● ● ●● ● ● ● ●●● ● ● ● ●● ● ● ● ● ●●● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ●● ●●● ● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ●● ● ● ● ● ● ● ● ●● ● ● ●● ● ● ● ● ●● ● ●● ●● ● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ●● ● ● ● ● ● ● ● ● ● ●● ● ● ● ●●● ● ● ● ●● ● ● ● ● ●●● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ●● ● ●● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ●● ● ● ● ● ● ● ● ● ●
● ● ● ● ● ● ● ● ●● ● ●● ●● ● ● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ●● ● ● ● ● ● ● ●● ● ● ●● ●● ● ● ●● ● ● ● ●● ● ●● ● ● ●● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ●● ● ●● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ●● ● ● ● ●● ● ● ● ●● ● ● ● ●●● ● ● ● ● ● ● ● ●● ● ●● ●● ● ● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ●● ● ● ● ● ● ● ●● ● ● ●● ●● ● ● ●● ● ● ● ●● ● ●● ● ● ●● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ●● ● ● ● ●● ● ● ● ●● ● ● ● ●●● ● ● ● ● ● ● ● ●● ● ●● ●● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ●● ● ● ● ● ● ● ●● ● ● ●● ●● ● ● ●● ● ● ● ●● ● ●● ● ● ●● ●● ● ●● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ●● ● ● ● ●● ● ● ● ●● ● ● ● ●●● ● ● ● ● ● ● ●● ● ●● ●● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ● ●● ● ● ●● ●● ● ● ●● ● ● ●● ● ●● ● ● ●●● ● ● ●● ● ● ● ●● ● ● ●● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ●● ●● ● ●● ● ● ● ●● ● ● ●●● ● ● ● ●● ● ● ●● ● ●● ●● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ●● ● ●● ● ● ●●● ● ● ●● ● ● ● ●● ● ● ●● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ●● ● ● ● ●● ● ● ●●● ● ● ● ●● ● ● ●● ● ●● ●● ● ●● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ● ● ●● ● ● ●● ● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ●●● ● ● ●● ● ● ● ●● ● ●● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ●● ● ● ● ●● ● ●● ●● ● ● ● ●● ● ● ●● ● ●● ●● ● ●● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ●●● ● ● ●● ● ● ● ●● ● ●● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ●● ● ● ● ●● ● ●● ●● ● ● ● ●● ● ● ●● ● ●● ●● ● ●● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ●●● ● ● ●● ● ● ● ●● ● ●● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ●● ● ● ● ●● ● ●● ●● ● ● ● ●● ● ● ●● ● ●● ●● ● ●● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ●●● ● ● ●● ● ● ● ●● ● ●● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ●● ● ● ● ●● ● ●● ●● ● ● ● ●● ● ● ●● ● ●● ●● ● ●● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●●● ● ● ●● ● ● ● ●● ● ●● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ●● ● ● ● ●● ● ●● ●● ● ● ● ●● ● ● ● ● ● ●● ●● ● ●● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ●● ● ●● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ● ● ● ● ●● ● ●● ●● ● ● ● ●● ● ● ● ● ● ●● ●● ●● ●● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ●● ● ●● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ● ● ● ● ●● ● ●● ●● ● ● ● ●● ● ● ● ● ● ●● ●● ● ●● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ●● ● ●● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ● ● ● ● ●● ● ●● ●● ● ● ● ●● ● ● ● ● ● ●● ●● ● ●● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ●● ● ●● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ● ● ● ● ●● ● ●● ●● ● ● ● ●● ● ● ● ● ● ●● ●● ● ●● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ●● ● ●● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ● ● ● ● ●● ● ●● ●● ● ● ● ●● ● ● ● ● ● ●● ●● ● ●● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ●● ● ●● ● ● ● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ● ● ● ● ●● ● ●● ●● ● ● ● ●● ● ● ● ● ● ●● ●● ● ●● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ●● ● ●● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ● ● ● ● ●● ● ●● ●● ● ● ● ●● ● ● ● ● ● ●● ●● ● ●● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ●● ● ●● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ● ● ● ● ●● ● ●● ●● ● ● ● ●● ● ● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ●● ● ●● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ● ● ● ● ●● ● ●● ●● ● ● ● ●● ● ● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ●● ● ● ● ●● ● ●● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●●● ● ●● ● ● ● ● ● ●● ● ●● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ●●● ● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ● ● ● ● ●● ● ●● ●● ● ●● ●● ● ● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ●● ● ●● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●● ●● ● ● ● ● ● ● ● ●● ● ●● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ●●● ● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ● ● ● ● ●● ● ●● ●● ● ●● ●● ● ● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ● ● ● ●● ●● ● ● ● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ●● ● ●● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●● ●● ● ● ● ● ● ● ● ●● ● ●● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ●● ● ●● ●●● ● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ● ● ● ● ●● ● ●● ● ●
● ●● ●● ● ● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ● ● ● ●● ● ●● ● ● ● ● ● ● ● ● ●● ● ● ● ●● ● ● ● ●● ● ●● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●● ●● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ●● ●● ● ● ●● ● ●● ●●● ● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ●● ● ●● ●● ● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ● ● ● ●● ● ●● ● ● ● ● ● ● ● ●●● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ●● ●●● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ●●● ●● ● ● ●● ● ●● ●●● ●● ● ● ●● ● ● ● ●● ● ● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ●● ●● ●●● ● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ● ● ●● ● ●● ● ● ● ● ● ● ● ●●● ● ● ● ● ●● ● ● ● ●● ● ●● ● ● ● ● ●● ● ●● ● ●● ● ● ● ● ● ● ● ●● ●●● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ●●● ●● ● ● ●● ● ●● ●●● ●● ● ● ●● ● ● ● ●● ● ● ● ● ● ●● ●● ● ●● ● ●● ● ● ● ● ● ● ● ● ● ●● ● ●● ●● ●●● ● ● ● ● ●● ● ● ● ●●● ● ● ● ● ● ● ●● ● ●● ● ● ● ● ● ● ● ●●● ● ● ● ● ●● ● ● ● ●● ● ●● ● ● ● ● ●● ● ●● ● ●● ● ● ● ● ●● ● ●● ●●● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ●●● ●● ● ● ●● ● ●● ●●● ●● ● ● ●● ● ● ● ●● ● ●● ● ● ●● ●● ● ●● ● ●● ● ● ● ● ● ● ● ● ● ●●● ●● ●● ●●● ● ● ● ● ●● ● ● ● ●●● ● ● ● ● ● ● ● ●● ● ●● ● ● ● ● ● ● ● ●●● ● ● ● ● ●● ● ● ● ●● ● ●● ● ● ●● ●● ● ●● ● ●● ● ● ● ● ●● ● ●● ●●● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ●●● ●● ● ● ●● ● ●● ●●● ●● ● ● ●● ● ● ● ●● ● ●● ● ● ●● ●● ● ●● ● ●● ● ● ● ● ● ● ● ● ● ●●● ●● ●● ●●● ● ● ● ● ●● ● ● ● ●●● ● ● ● ● ● ● ● ●● ● ●● ● ● ● ● ● ● ● ●●● ● ● ● ● ●● ● ● ● ●● ● ●● ● ● ●● ●● ● ●● ● ●● ● ● ● ● ●● ● ●● ●●● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ●●● ●● ● ● ●● ● ●● ●●● ●● ● ● ●● ● ● ● ●● ● ●● ● ● ●● ●● ● ●● ● ●● ● ● ● ● ● ● ● ● ● ●●● ●● ●● ●●● ● ● ● ● ●● ● ● ● ●●● ● ● ● ● ● ● ● ●● ● ●● ● ● ● ● ● ● ● ●●● ● ● ● ● ●● ● ● ● ●● ● ●● ● ● ●● ●● ● ●● ● ●● ● ● ● ● ●● ● ●● ●●● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ●●● ●● ● ● ●● ● ●● ●●● ●● ● ● ●● ● ● ● ●● ● ●● ● ● ●● ●● ● ●● ● ●● ● ● ● ● ● ● ● ● ● ●●● ●● ●● ●●● ● ● ● ● ●● ● ● ● ●●● ● ● ● ● ● ● ● ●● ● ●● ● ● ● ● ● ● ● ●●● ● ● ● ● ●● ● ● ● ●● ● ●● ● ● ● ● ●● ● ●● ● ●● ● ●● ● ●● ● ●● ●●● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ●●● ●● ● ● ●● ● ● ● ●●● ●● ● ● ●● ● ● ● ●● ● ●● ● ● ● ● ● ● ● ●● ● ●● ● ● ● ● ● ● ● ● ● ●●● ●● ●● ●●● ● ●● ● ●● ● ● ● ●●● ● ● ● ● ● ● ● ●● ● ●● ● ● ● ● ● ● ● ●●● ● ● ● ● ●● ● ● ● ●● ● ●● ● ● ● ● ●● ● ●● ● ●● ● ●● ● ●● ● ●● ●●● ● ●● ● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ●●● ●● ● ● ●● ● ● ● ●●● ●● ● ● ●● ● ● ● ●● ● ●● ● ● ● ● ● ● ● ●● ● ●● ● ●● ● ● ● ● ● ● ●●● ●● ●● ●●● ● ●● ● ●● ● ● ● ●●● ● ● ● ● ● ● ● ●● ● ●● ● ● ● ● ● ● ● ●●● ● ● ● ● ●● ● ● ● ●● ● ●● ● ● ● ● ●● ● ●● ● ●● ● ●● ● ●● ● ●● ●●● ● ●● ● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ●●● ●● ● ● ●● ● ● ● ●●● ●● ● ● ●● ● ● ● ●● ● ●● ● ● ● ● ● ● ● ●● ● ●● ● ●● ● ● ● ● ● ● ●●● ●● ●● ●●● ● ●● ● ●● ● ● ● ●●● ● ● ● ● ● ● ● ●● ● ●● ● ● ● ● ● ● ● ●●● ● ● ● ● ●● ● ● ● ●● ● ●● ● ● ● ● ●● ● ●● ● ●● ● ●● ● ●● ● ●● ●●● ● ●● ● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ●●● ●● ● ● ●● ● ● ● ●●● ●● ● ● ●● ● ● ● ●● ● ●● ● ● ● ● ● ● ● ●● ● ●● ● ●● ● ● ● ● ● ● ●●● ●● ●● ●●● ● ●● ● ●● ● ● ● ● ●● ● ● ● ● ● ● ● ●● ● ●● ● ● ● ● ● ● ● ●●● ● ● ● ● ●● ● ● ● ●● ● ●● ● ● ● ● ●● ● ●● ● ●● ● ● ● ● ●● ● ●● ●●● ● ●● ● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ●●● ●● ● ● ● ● ● ● ● ●●● ●● ● ● ●● ● ● ● ●● ● ●● ● ● ● ● ● ● ● ●● ● ●● ● ●● ● ● ● ● ● ● ●●● ●● ●● ●●● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ● ● ● ●● ● ●● ● ● ● ● ● ● ● ●●● ● ● ● ● ●● ● ● ● ●● ● ●● ● ● ● ● ●● ● ●● ● ●● ● ● ● ● ●● ● ●● ●●● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ●●● ●● ● ● ● ● ● ● ● ●●● ●● ● ● ●● ● ● ● ●● ● ●● ● ● ● ● ● ● ● ●● ● ●● ● ● ● ● ● ● ● ● ● ●●● ●● ●● ●●● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ● ● ● ●● ● ●● ● ● ● ● ● ● ● ●●● ● ● ● ● ●● ● ● ● ●● ● ●● ● ● ● ● ●● ● ●● ● ●● ● ● ● ● ● ● ● ●● ●●● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ●●● ●● ● ● ●● ● ● ● ●● ● ●● ● ● ● ● ● ● ● ●● ● ●● ● ● ● ● ● ● ● ● ● ●●● ●● ●● ●●● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ● ● ● ●● ● ●● ● ● ● ● ● ● ● ●●● ● ● ● ● ●● ● ● ● ●● ● ●● ● ● ● ● ●● ● ●● ● ●● ● ● ● ● ● ● ● ●● ●●● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ●●● ●● ● ● ● ● ● ● ● ●●● ●● ● ● ●● ● ● ● ●● ● ●● ● ● ● ● ● ● ● ●● ● ●● ● ● ● ● ● ● ● ● ● ●●● ●● ●● ●●● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ● ● ● ●● ● ●● ● ● ● ● ● ● ● ●●● ● ● ● ● ●● ● ● ● ●● ● ●● ● ● ● ● ●● ● ●● ● ●● ● ● ● ● ● ● ● ●● ●●● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ●●● ●● ● ● ● ● ● ● ● ●●● ●● ● ● ●● ● ● ● ●● ● ●● ● ● ● ● ● ● ● ●● ● ●● ● ● ● ● ● ● ● ● ● ●●● ●● ●● ●●● ● ● ● ● ●● ●● ● ● ●● ● ● ● ● ● ● ● ●● ● ●● ● ● ● ● ● ● ● ●●● ● ● ● ● ●● ● ● ● ●● ● ●● ● ● ● ● ●● ● ●● ● ●● ● ● ● ● ● ● ● ●● ●●● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ●●● ●● ● ● ● ● ● ● ● ●●● ●● ● ● ●● ● ● ● ●● ● ●● ● ● ● ● ● ● ● ●● ● ●● ● ● ● ● ● ● ● ● ● ●●● ●● ●● ●●● ● ● ● ● ●● ●● ● ● ●● ● ● ● ● ● ● ● ●● ● ●● ● ● ● ● ● ● ● ●●● ● ● ● ● ●● ● ● ● ●● ● ●● ● ● ● ● ●● ● ●● ● ●● ● ● ● ● ● ● ● ●● ●●● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ●●● ●● ● ● ● ● ● ● ● ●●● ●● ● ● ●● ● ● ● ●● ● ●● ● ● ● ● ● ● ● ●● ● ●● ● ● ● ● ● ● ● ● ● ●●● ●● ●● ●●● ● ● ● ● ●● ●● ● ● ●● ● ● ● ● ● ● ● ●● ● ●● ● ● ● ● ● ● ● ●●● ● ● ● ● ●● ● ● ● ●● ● ●● ● ● ● ● ●● ● ●●● ●● ● ● ● ● ● ● ● ●●● ●● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ●●● ●● ● ● ● ● ● ● ● ●●● ●● ● ● ●● ● ● ● ●● ● ●● ● ● ● ● ● ● ● ●●● ●● ● ● ● ● ● ● ● ● ● ●●● ●● ●● ● ●● ● ● ● ● ●● ●● ● ● ●● ● ● ● ● ● ● ● ●● ● ●● ● ● ● ● ● ● ● ●●● ● ● ● ●● ● ● ● ●● ● ●● ● ● ● ● ●● ● ●●● ●● ● ● ● ● ● ● ● ●●● ●● ● ●● ● ● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ●●● ●● ● ● ● ● ● ● ● ●●● ●● ● ● ●● ● ● ● ●● ● ●● ● ● ● ● ● ● ● ●●● ●● ● ● ● ● ● ● ● ● ● ●●● ●● ●● ● ●● ● ● ● ● ●● ●● ● ● ●● ● ● ● ● ● ● ● ●● ●● ● ● ● ● ● ● ●●● ● ● ● ●● ● ● ● ●● ● ●● ● ● ● ●● ● ●●● ●● ● ● ● ● ● ● ● ●●● ●● ● ● ● ● ● ● ●● ● ● ●● ● ● ● ● ● ● ● ●●● ●● ● ● ● ● ● ● ● ●●● ●● ● ● ●● ● ● ● ●● ● ●● ● ● ● ● ● ● ● ●●● ●● ● ● ● ● ● ● ●● ● ● ●● ● ●● ● ● ●● ● ● ● ● ●● ●● ● ● ●● ● ● ● ● ● ● ● ● ●● ●● ● ● ● ● ● ● ● ● ●● ● ●● ● ●● ● ● ● ● ● ● ●● ● ● ● ● ●● ● ●●● ●● ● ● ● ● ● ● ● ● ●�