Proceedings of the 15th IBPSA ConferenceSan Francisco, CA, USA, Aug. 7-9, 2017

848https://doi.org/10.26868/25222708.2017.224

CALIBRO: an R Package for the

Automatic Calibration of Building Energy Simulation Models

Filippo Monari, Paul StrachanUniversity of Strathclyde, Glasgow, UK

Abstract

Bayesian probability theory offers a powerful frame-work for the calibration of building energy models(Bayesian calibration). The major issues impedingits routine adoption are its steep learning curve, andthe complicated setting up of the required calcula-tion. This paper introduces CALIBRO, an R packagewhich has the objective of facilitating the undertakingof Bayesian calibration of building energy models. Anoverview of the techniques and procedures involved inCALIBRO is given, as well as demonstrations of itscapability and reliability through two examples.

Introduction

Previous research has highlighted a significant gap be-tween building energy performance predicted in thedesign stage and that achieved in the operationalstage. This issue may be addressed by ensuring thatbuilding energy models are equally applicable to theoperation stage by reconciling their predictions withfield observations. Recently, significant research ef-fort has been focused on the development of meth-ods for tuning model parameters in order to im-prove predictions, that is the calibration of Build-ing Energy Simulation (BES) models against mea-sured data. A novel approach is to employ probabilis-tic emulators based on Gaussian Process Regression(GPR) (Rasmussen and Williams (2006)) in a quasi-Bayesian framework (Bayesian calibration). Such anapproach has several advantages with respect to pre-viously employed methodologies like manual iterativeprocedures and the employment of optimisation tech-niques.Manual iterative procedures have been largely appliedin the past (Clarke et al. (1993), Pedrini et al. (2002),Yoon et al. (2003), Westphal and Lamberts (2005),Raftery et al. (011a)) and Raftery et al. (011b). Theirhigh dependence on the skills and expertise of theanalyst makes them very heterogeneous. This factortogether with the absence of a strong mathematicalframework has led to criticisms about the consistencyof this approach. The main issue is the inadequateconsideration of modelling uncertainties.In few recent studies (Reddy et al. (2007a), Reddyet al. (2007b), Sun and Reddy (2006), Johnson andHu (2012) and Coakley et al. (2011) optimisationtechniques, especially Monte Carlo techniques, havebeen used to aid the calibration of BES models. Thisapproach follows a more rigorous mathematical basis.

However the consideration of modelling uncertaintiesis not satisfactory. Only a few solution vectors, notsufficient to characterise the uncertainties related toparameter estimates, are provided.

Bayesian calibration improves on the two approachesmentioned, by offering the opportunity to treat thecalibration problem probabilistically (Kennedy andO’Hagan (2001)). It allows for a comprehensive treat-ment of the uncertainties (observation errors, pa-rameter uncertainties and model deficiencies), whichare consistently considered throughout the calibra-tion process. This influences the solution, which isreturned in the form of full posterior probability den-sity distributions, reflecting the uncertainties aboutparameter estimates. Furthermore, the Bayesianparadigm allows the rigorous inclusion of modellerknowledge in the calculations, through the modelstructure and parameters’ prior probability densitydistributions. For these reasons, Bayesian calibrationcan be applied effectively for the calibration of BESmodels under uncertainties as shown in Heo et al.(2012), Heo et al. (2013), Kim et al. (2014) and Heoet al. (2015).

The central problems impeding its widespread adop-tion, are its steep learning curve and the absence ofmeans for its routine application. CALIBRO is anR package which aims to overcome these problemsby facilitating the setting up and the undertaking ofthe calculations underpinning the Bayesian analysisof calibration problems, thus allowing the automaticBayesian calibration of BES models.

CALIBRO

As far as the authors are aware, CALIBRO is the firstsoftware package dedicated to the Bayesian calibra-tion of BES models. CALIBRO has been based uponthe mathematical framework established in Monari(2016), which builds on the methods depicted inKennedy and O’Hagan (2001), Bayarri et al. (2005),and especially Higdon et al. (2008). The main us-age of CALIBRO is envisaged to be the correctionof models built at the design phase, by tuning theirparameters according to information gained throughoperational monitoring.

CALIBRO uses a black-box approach and can cali-brate any building energy model. The information itrequires is:

• measurements of the observed performance indi-cators;

Proceedings of the 15th IBPSA ConferenceSan Francisco, CA, USA, Aug. 7-9, 2017

849

• measurements of the boundary conditions influ-encing the observed performance indicators;

• a sample of model outputs, predicting the ob-served performance indicator, and the corre-sponding model inputs.

This set of information is called a calibration dataset.Different calibration datasets can be used in the samecalibration thus considering different performance in-dicators and boundary conditions simultaneously. Itis advised to carry out the model simulations accord-ing to Latin Hyper Cube design of the model inputs.The parameters boundaries should be established ac-cording to a previous uncertainty analysis. A samplesize of about 10 times the number of model inputs issuggested.

Figure 1 gives an overview of the steps and techniquescombined in CALIBRO.

At first, in order to simplify and speed up the calcula-tions, the dimensionality of the calibration dataset isreduced by applying Principal Component Analysis(Jolliffe (2002)) as in Higdon et al. (2008). Struc-tural identifiability is then solved by means of a vari-ance based sensitivity analysis. The method in Rattoand Pagano (2010) is applied as indicated in Lamboniet al. (2011) to automatically select the most influen-tial parameters (calibration parameters). This tech-nique allows the calculation of the estimates of theparameters’ Sobol’ first order indices (Si) and of therelative standard errors (se). A parameter is deemedto be influential if its Si is significantly higher than0 (i.e. > 0 with 95% probability assuming Si nor-mally distributed with mean the calculated estimateand standard deviation the corresponding standarderror).

In the subsequent phase Gaussian Process Regres-sion in a quasi-Bayesian framework is used to builda black box emulator of the BES model, consideringonly the calibration parameters. This emulator is op-timised with the Nelder-Mead method (Nelder andMead (1965)). The training is considered successfulif a Determination Coefficient at least equal to thesum of the calibration parameters’ Si is achieved.

Subsequently, the trained emulator is used to inferthe calibration parameters’ joint posterior probabil-ity density distribution (Monari (2016)), which is in-tegrated with the Adaptive Metropolis-within-Gibbsalgorithm (Rosenthal (2007)) to find the posteriorprobability density distribution for each calibrationparameter. The convergence of the MCMC simula-tions is monitored with the R package CODA (Plum-mer et al. (2006)). When only one chain is simulated,its convergence is assessed depending on the ratiobetween the chain mean and the relative correctedstandard error being smaller than a given threshold.When multiple chains are run in parallel, the Gelmanand Rubin’s convergence diagnostic (Gelman and Ru-bin (1992)) is employed. Finally the calibration pa-rameters’ estimates, consisting of the Maximum A

Posteriori (MAP) values and the relative confidenceintervals are returned.

Figure 1: steps and techniques involved in CALIBRO.

Experiments

In the following two examples of CALIBRO applica-tions are described. Two very different models arecalibrated in order to demonstrate the flexibility ofthe method implemented in the calibration softwarepackage. The first model is of a single building com-ponent, namely a multilayer wall. The second modelrepresents a small domestic building, and takes intoaccount complex phenomena like wind driven infil-tration. For both of these case studies, two kinds ofexperiments are described:

• virtual experiments: where the models are cali-brated against synthetic observations for whichthe true parameter values are known, in order totest the correct implementation of the method;

• real experiments: where the model is calibratedagainst the measured observations, in order totest practical applications of CALIBRO.

The wall

The first is the calibration of a model aiming to re-produce the measured heat flux through a multilayerwall of a laboratory in an insulation factory in thesouth of Sweden. The experiment was conductedby the EC Joint Research Centre, Institute for En-ergy and Transport in ISPRA, Italy, and consisted ofmonitoring the heat flux through the wall, and thetemperatures at the internal and external surfaces,

Proceedings of the 15th IBPSA ConferenceSan Francisco, CA, USA, Aug. 7-9, 2017

850

for a period of one month. Data with a 30 minutestime step were collected. The construction elementhad three layers: a central core of gas concrete blocks(GC) of thickness 150 mm and insulation glass fibreboards (FB) of thickness 27 mm at both sides. Atthe end of the experiment, samples were taken fromthe test component in order to determine the proper-ties of each material. For FB, values for density andconductivity of 116.6 kg/m3 and 31.27 mW/(mK)respectively were specified. For GC, only the density(552 kg/m3) was provided.The model to be calibrated consisted of two ther-mal zones, one representing the inside of the labo-ratory, and the other depicting the exterior environ-ment, divided by the test wall. The boundary condi-tions, namely the internal and external temperatures,were imposed on the two faces of the test componentthough suitable control laws. A three layer construc-tion, reproducing the given specifications, was usedto represent the test wall layer structure. The un-known model parameters consisted of gas concreteblock conductivity (GCk), gas concrete block specificheat (GCc) and glass fibre board specific heat (FBc).The initial values and prior probability density distri-butions assumed for these parameters are indicatedin Table 1. A set of 30 model input-output pairs wasused as input for CALIBRO.

Table 1: The Wall: calibration parameters’ priorprobability density distributions and initial values.

PARA- INITIAL PRIORMETER VALUES DIST

GCk ( WmK ) 0.12 Uniform(0.05, 0.15)

GCc ( JkgK ) 800.00 Uniform(600, 1000)

FBc ( JkgK ) 800.00 Uniform(600, 1000)

In Table 2 are reported the results of the sensitivityanalysis phase under the form of estimated fractionsof model output variance attributable to each modelparameter by itself (Si). GCk clearly dominated themodel, GCc had only a marginal role, while FBc,having a negligible effect, was deemed not identifiableand it was not considered as a calibration parameterby CALIBRO.

Table 2: The wall: parameters’ main effects (Si) andrelative standard errors (se).

PARAMETER Si seGCk 0.954 0.007GCc 0.043 0.070FBc 0.002 0.018

Since this example required a relatively short com-putational time, at first, a series of virtual tests wasundertaken to demonstrate the performance of CAL-IBRO depending on different levels of noise in thedata. Synthetic target observations were generated

for the initial values in Table 1 and then Gaussianwhite noise was added according to 34 increasingnoise to variance ratios (NVRs). Ten calibrationswere carried out, by simulating single Markov chains,for each NVR for a total of 340. In order to performthe analysis in a reasonable time a relatively high con-vergence tolerance was used (0.002). Figures 2 and3 summarise the results of this set of trials. Herethe blue box plots represent the samples of 0.025 and0.975 quantiles calculated for each NVR. Similarly,the black box plots indicate the distribution of cali-bration parameter estimates. Blue and black dots arethe average values resulting from considering only twosignificant figures.

Figure 2: The wall–virtual experiments: GCk calibra-tion results for different NVRs. SF indicates Signifi-cant Figures.

Figure 3: The wall–virtual experiments: GCc calibra-tion results for different NVRs. SF indicates Signifi-cant Figures.

Subsequently, the real observations were considered.The calibration parameters were inferred by runningthree different Markov chains in parallel. Figures 4and 5 show the evolution of mean, 0.025 and 0.975quantiles for such chains during the simulation, re-spectively for GCk and GCc. The calibration resultsare summarised in Table 3, and a comparison between

Proceedings of the 15th IBPSA ConferenceSan Francisco, CA, USA, Aug. 7-9, 2017

851

calibrated model prediction and observations is de-picted in Figure 6.

Figure 4: The wall–real experiment: evolution ofthree Markov chains for GCk: traces and summarystatistics.

Figure 5: The wall–real experiment: evolution ofthree Markov chains for GCc: traces and summarystatistics.

Figure 6: The wall–real experiment: comparison be-tween calibrated model predictions (black line) and ob-servations (red line). The 95% confidence intervalsare indicated in grey.

Table 3: The Wall–real experiment: calibration pa-rameters estimates and 95% confidence intervals.

PARA- MAP q0.025 q0.5 q0.975METER

GCk ( WmK ) 0.118 0.113 0.118 0.122

GCc ( JkgK ) 993 890 971 995

The house

In the second example the object of the analysis wasto calibrate a model aiming to predict the ground andupper floor average temperatures in a small domes-tic building (Figure 7), which was used as test facil-ity in developing model predictive control systems.The building is located at the BRE Innovation Parkin Motherwell (Scotland), and it is a prototype fora modular mass market low energy house design. Itsmain heat source is 5 kW exhaust air heat pump feed-ing underfloor heating. The unit also includes a smallhot water store. A 1 kW array of photovoltaic pan-els is located on the roof. Being a test facility, thehouse was unoccupied, and its properties were exten-sively investigated through dedicated tests performedby the research team. A blower door test was under-taken to characterise air tightness, and its thermalbehaviour was observed subject to dedicated thermalpulse testing, and to a normal heating regime. Themonitoring period lasted approximately one month,collecting data with a 10 minute time step. The datacomprised external temperature, wind velocity, winddirection, relative humidity, diffuse solar radiation,direct solar radiation, heat injected in the groundfloor and heat injected in the first floor. This datasetwas used by the research team to manually calibratea BES model. The results that are presented in thefollowing have been achieved by applying CALIBROindependently afterwards, with the purpose of evalu-ating the eventual benefit of its employment in futurestudies.

Figure 7: The house.

One virtual experiment and two real experiments

Proceedings of the 15th IBPSA ConferenceSan Francisco, CA, USA, Aug. 7-9, 2017

852

(real 1 and real 2) were undertaken. In the virtual andreal 1 experiments, 55 parameters were considered,consisting of conductivities, densities, window ther-mal resistances, crack lengths, and constant volumeflow rates. Construction properties (conductivities,densities, window thermal resistances) were fairly wellknown, while much more uncertain were deemed tobe the airflow network parameters (crack lengths, andconstant volume flow rates). Therefore the formerwere varied within ±15% and the latter were changedwithin ±80% of their design values according to uni-form probability density distributions. Subsequently(real 2 experiment), windows optical transmissionswere included in the calibration parameters, since cor-relation was noticed between the solar radiation andthe residuals between model predictions and observa-tions. These were varied within ±90% of their designvalues. A sample of 550 model input-output pairs wasused as input for both the analyses.

Virtual and real 1 experiments

The outcomes from the sensitivity analysis showedthat the model was highly conditioned by the valueof the parameter HPext which represents the volumeflow rate of the heat pump exhaust air extraction.However, by breaking down the sensitivity results ac-cording to the principal components, CALIBRO iden-tified other minor effects from some parameters rep-resenting lengths of cracks around the frames of southand west windows located at the ground floor of thebuilding (crck2l and crck3l). This group of model in-puts was considered as calibration parameters. Theirsensitivity indexes are displayed in Table 4, and therelative initial values and prior probability densitydistributions are listed in Table 5.

Table 4: The house - virtual and real 1 experiments:estimates of parameters’ main effects (Si) and rela-tive standard errors (se).

PARAMETER Si seHPext 0.965 0.001crck2l 0.006 0.003crck3l 0.006 0.005

Table 5: The house - virtual and real 1 experiments:calibration parameters’ prior probability density dis-tributions and initial values.

PARA- INITIAL PRIORMETER VALUES DIST

HPext ( m3

103s ) 4.3 Uniform(0.86, 7.74)crck2l (m) 4.6 Uniform(0.92, 8.28)crck3l (m) 6.2 Uniform(1.24, 11.16)

Firstly the model was calibrated against synthetic ob-servations, which were obtained by running the modelfor the initial values in Table 5. In this case nonoise was added, in order to assess the capabilities

of the software of identifying very weak parameterslike crck2l and crck3l. The evolutions of the threeMarkov chains used in this calibration are shown inFigures 8, 9 and 10.

Figure 8: The house - virtual experiment: evolutionof three Markov chains for HPext: traces and sum-mary statistics.

Figure 9: The house - virtual experiment: evolutionof three Markov chains for crck2l: traces ans sum-mary statistics.

After having substituted the synthetic observationswith the measured ground floor (GF) and upper floor(UF) temperatures, a new calibration was under-taken. The results are summarised in Table 6 contain-ing calibration parameters’ estimates and confidenceintervals.

Table 6: The house - real 1 experiment: calibrationparameters estimates and 95% C.I..

PARA- MAP q2.5 q50 q97.5METER

HPext ( m3

103s ) 4.53 4.5 4.53 4.55crck2l (m) 4.11 4.10 4.16 4.37crck3l (m) 5.63 5.62 5.78 6.29

Proceedings of the 15th IBPSA ConferenceSan Francisco, CA, USA, Aug. 7-9, 2017

853

Figure 10: The house - virtual experiment: evolutionof three Markov chains for crck3l:traces and summarystatistics.

Real 2 experiment

In this experiment the the optical transmission ofthe ground floor windows (TrGF ) and the opticaltransmission of the upper floor windows (TrUF ) wereadded to the free parameters of the model, after hav-ing noticed correlation between the solar radiationand the residuals between observed temperatures andpredictions of the model calibrated in the real 1 ex-periment. The outcomes from the sensitivity analy-sis step (Table 7) highlighted that TrGF and TrUF

had effects comparable to the main parameter’s one(HPext).

Table 7: The house - real 2 experiment: estimates ofcalibration parameters’ main effects (Si) and relativestandard errors (se).

PARAMETER Si seTrGF 0.14 0.041TrUF 0.39 0.045HPext 0.57 0.037crck2l 0.02 0.016crck3l 0.005 0.009

Table 8: The house - real 2 experiment: optical trans-mission parameters’ prior probability density distribu-tions and initial values.

PARA- INITIAL PRIORMETER VALUES DISTTrGF (-) 0.346 Uniform(0.01, 0.66)TrUF (-) 0.346 Uniform(0.01, 0.66)

The calibration was undertaken by running threeMarkov chains in parallel. Their evolutions for thethree most significant calibration parameters (TrGF ,TrUF and HPext) are summarised in Figures 11, 12and 13. The inferred estimates and confidence inter-vals are contained in Table 9. The model calibrated

in this test had the best predictive performances (Ta-ble 10). The achieved matches between its predictionsand measured temperatures are graphically describedin Figure 14 and 15.

Figure 11: The house - real 2 experiment: evolutionof three Markov chains for TRGF : traces and sum-mary statistics.

Figure 12: The house - real 2 experiment: evolutionof three Markov chains for TRUF : traces and sum-mary statistics.

Table 9: The house - real 2 experiment: calibrationparameters estimates and 95% C.I..

PARA- MAP q2.5 q50 q97.5METERTrGF (-) 0.345 0.154 0.34 0.57TrUF (-) 0.101 0.001 0.080 0.367

HPext ( 103m3

s ) 3.10 2.17 2.92 5.52crck2l (m) 4.43 1.22 4.29 8.08crck3l (m) 3.02 1.32 2.66 6.83

Discussion and result analysisThe undertaken virtual experiments demonstratedthe capability of CALIBRO to identify effectively cali-bration parameter values, even when significant levelsof noise contaminate the measurements.

Proceedings of the 15th IBPSA ConferenceSan Francisco, CA, USA, Aug. 7-9, 2017

854

Figure 13: The house - real 2 experiment: evolutionof three Markov chains for HPext: traces and sum-mary statistics.

Figure 14: The house - real 2 experiment: compari-son between ground floor temperature predictions andobservations.

Figure 15: The house - real 2 experiment: compari-son between upper floor temperature predictions andobservations.

In the first series of virtual trials (Figures 2 and 3),CALIBRO was always able to correctly estimate theconsidered model inputs even for high noise to vari-

ance ratios. Indeed, despite little inaccuracy, thatmay be due to the premature stop of the MCMCsimulations because of the high convergence thresh-old adopted, the true parameters values are alwayswell centred within the calculated confidence inter-vals and close to the inferred estimates. If the effectsof such small inaccuracies are diminished by roundingthe results to suitable precision, that is two significantfigures all the calibrated values are equal to the actualparameter values.Similar conclusions can be drawn from the outcomesof the calibration against synthetic observations per-formed on the house model. All the MCMC sim-ulation converged to the same stationary distribu-tion (Figures 8, 9 and 10) and all the calibration pa-rameters were estimated correctly. HPext has been

negligibly overestimated (0.0044 ≈ 0.0043m3

s ) andeven crck2l and crck3l, although having almost in-discernible effects, were well identified. In particu-lar, their empirical probability density distributions,despite having large variances, are clearly peakedaround the respective true values (Figures 16 and 17).These last results are consistent with those depictedin Figure 2 and 3. As indicated by the box plots,the calculated values become increasingly uncertainwith the noise. Similarly the average confidence in-tervals broaden as the NVR grows, especially for GCc

which is the weakest of the two calibration parame-ters. Thus, measurement uncertainties are correctlyinfluencing the calculations, and the results correctlyfollow the sensitivity of the model, returning higheruncertainties for the less important parameters.

Figure 16: The house–virtual experiment :crck2l em-pirical probability density distribution.

The performed real experiments proved the abilityof CALIBRO to tune a model’s inputs in order toimprove the efficacy of such a model in predicting ob-served performance indicators. It is difficult to judgethe actual correspondence between the estimated pa-rameters and the real properties of the wall that theyrepresent, since the specifications of the wall have notbeen disclosed, while airflow network parameters, forthe house experiment, are naturally unknown due to

Proceedings of the 15th IBPSA ConferenceSan Francisco, CA, USA, Aug. 7-9, 2017

855

Figure 17: The house–virtual experiment: crck3l em-pirical density distribution.

the difficulty of measuring them. However, confidencein the goodness of the calibration results is given byFigures 4, 5, 11, 12 and 13, which show that duringthe calibrations of the model against real measure-ments all the MCMC simulations converged to thesame probabilistic solutions. Especially the strongestparameters (GCk, HPext, TRGF , and TRUF ) showedreasonable estimates well centred in the respectiveconfidence intervals, and characterised by bell shapedempirical probability density distributions, indicat-ing that these variables were effectively estimated.Conversely, the weakest parameters (GCc, crck2l andcrck3l) showed estimates close to their confidenceboundaries and large variances. Probably, their es-timations were influenced by observation errors andmodel deficiencies. In this case CALIBRO tends toexcessively modify the least important model param-eters resulting in a little over-fit of the measurements,as discussed more in detail in Monari (2016). Never-theless, due to their minor effects, errors in the esti-mates of these parameters do not affect sensibly themodel predictions.As shown by the graphical comparison in Figure 6, itwas possible to achieve a very good match betweenthe heat flux through the wall predicted by the cali-brated model and that measured. In particular, it isnot possible to notice any significant discrepancy be-tween predictions and measurements, which are con-sistently with the 95% confidence intervals. This isfurther highlighted by the low Root Mean Square Er-ror (RMSE), equal to 0.1 W/m2.The final calibrated model for the house was able to fitwell the mean temperature of the ground floor (Figure14), while it showed some inaccuracies in predictingthe average temperature of the upper floor (Figure15). In particular the predicted upper floor temper-ature seems too sensitive to heat gains from solar ra-diation and thermal pulses. It is interesting to noticethe two different inferred values for the window opti-cal transmission. TrUF had an MAP value less thanone third of TrGF . Different window specificationsfor the two floors are deemed to be unlikely, and such

difference may be due to accumulated dirt or somephenomena not recorded during the monitoring. Amore likely explanation might be a lack of thermalmass in the upper floor. Material densities were nothighlighted by the sensitivity analyses performed assignificant parameters. The cause may be their in-dividual consideration instead of taking into accountmore global parameters governing the building ther-mal capacity. This may suggest that the inclusionof details in the model should be done gradually, de-pending on the outcome of the calibration. In partic-ular it is believed that, at first, it would be convenientto consider few important macro-parameters actingon different locations of the model, and then breakdown such macro-parameters according on the resultsof analysis. The correction of this model deficiencyrequires more investigation which will be object offuture studies. For the purpose of this research it ismore useful to compare the manual iterative calibra-tion process and its outcomes with the application ofCALIBRO and its results.

The time spent to manually calibrate the model wastwo days. The process involved the empirical identifi-cation of the calibration parameters according to theexpertise and knowledge of the modeller, and theiriterative variation until a satisfying match with theobserved temperatures was achieved. In the sameamount of time it was possible to apply CALIBROtwice. The most time consuming phases were gener-ating the required samples of simulations, that werecarried out overnight. The actual calibrations re-quired a few hours, during which the identificationof calibration parameters was performed automat-ically through a detailed variance based sensitivityanalysis, and the identification of a solution was per-formed according to robust probabilistic techniquesallowing the assessment of the reliability of the in-ferred model parameter estimates. It should also benoted that a relatively short lasting manual calibra-tion process, as in this case, is more an exception, dueto the large amount of information available, than arule. Furthermore, if the analyst was willing to per-form the selection of the calibration parameters basedupon sensitivity analysis, the required time wouldhave increased greatly. Finally, both the models cali-brated with CALIBRO achieved better matches withthe measurements (Table 10), having overall 34% and55% lower RMSEs.

Table 10: The house–real experiment: RMSE.

CALIBRATION GF UF GLOBALTYPE

manual iterative 0.69 1.72 1.31CALIBRO - real 1 0.60 1.07 0.86CALIBRO - real 2 0.62 0.81 0.72

Proceedings of the 15th IBPSA ConferenceSan Francisco, CA, USA, Aug. 7-9, 2017

856

Conclusions

This paper has introduced CALIBRO, an R pack-age for the Bayesian calibration of building energymodels. Such a software package allows the easy un-dertaking of the calibration of BES models accordingto a Bayesian framework, providing the analysts withthe possibility of rigorously introducing their knowl-edge in the calculations, and to prove or disprove theirhypothesis against the information provided by fieldobservations. The main features of CALIBRO are thefollowing.

• Model independence: adopting a black-box ap-proach, the calibration process can be appliedto any building energy model, requiring only theinformation defined in the calibration dataset asinputs.

• Multi-objective calibration: multiple perfor-mance indicators can be contained in one cali-bration dataset, and in turn multiple calibrationdatasets can be considered simultaneously dur-ing the calibration.

• Sensitivity analysis: the selection of the calibra-tion parameters is performed automatically ac-cording to a detailed variance based sensitivityanalysis.

• Robust probabilistic and statistical techniques:the inference of the calibration parameter val-ues most likely to reduce the gap between modelpredictions and field measurements is under-taken though optimisation and MCMC algo-rithms, which provide assessment of the reliabil-ity of the calculated estimates.

The capabilities of CALIBRO have been demon-strated in the described virtual and real experiments.The results relative to the former class of tests,showed good performance in estimating calibrationparameters even if the effects of such parameters onthe model output are small, and also when signifi-cant noise contaminates the measurements. The in-ferred calibration parameter estimates were reliable,reflecting in their uncertainties the magnitude of theobservation errors and the sensitivity of the model.The outcomes from the real examples, demonstratedthat CALIBRO is effective at improving the predic-tive capabilities of BES models. With respect to thehouse example, the model calibrated with CALIBROproduced better fits of the measured internal tem-peratures than that manually calibrated. However,this does not mean that the application of the soft-ware package will yield systematically a better fitwith respect to other approaches. The assessmentof its performance in comparison to other kinds ofcalibration procedure would require a statistical in-vestigation which could be object of future research.Nevertheless, the calibration of the house model withCALIBRO required less time than the manual cali-bration of the same model. Furthermore, the outputs

from CALIBRO could be used to asses the reliabilityof calculated calibration parameter values, buildingup confidence in the analyst that a good solution wasfound. The information returned by CALIBRO isalso effective in spotting model deficiencies and conse-quent model improvements, thus making the softwarepackage a useful tool for the analysis and diagnosis ofBES models against monitored data. Therefore theauthors are confident in saying that, compared to theusual practice of manually calibrating building en-ergy models, the employment of CALIBRO offers atheoretical edge, achieving more reliable results in ashorter period of time.However, the software has limitations that requireimprovements. Calibration parameters are identifiedonly according to the structural identifiability, whileit would be important to take into account the ex-perimental identifiability as well. It is currently im-possible to consider vectorial model parameters, suchas pressure coefficients, time varying convection coef-ficients, unknown heat gains or occupancy schedules,which are an important class of inputs for BES mod-els. Also a development priority is to provide a meansto compare different calibrated models through reli-able model selection criteria.CALIBRO has completed a first development phase,and it is now undergoing a testing stage in the contextof Hit2Gap European project (Costola et al. (2017)).The authors are confident that its robustness and re-liability will be further strengthened by this testingprocess. A first version of the software can be foundat http://www.esru.strath.ac.uk/software.htm.The final purpose is to provide, with CALIBRO, a so-phisticated tool supporting the development of novelpractices and systems, advancing Building EnergySimulation and allowing its routine application dur-ing the operational phase of a building life cycle.

ReferencesBayarri, M. J., J. O. Berger, D. Higdon, M. C.

Kennedy, A. Kottas, R. Paulo, J. Sacks, J. A.Cafeo, J. Cavendish, C. H. Lin, and J. Tu (2005).A framework for validation of computer models.Technical report, National Institute of Statisti-cal Sciences, 19 T. W. Alexander Drive, PO Box14006, Research Triangle Park, NC 27709-4006,www.niss.org.

Clarke, J. A., P. A. Strachan, and C. Pernot (1993).An approach to the calibration of building energysimulation models. ASHRAE Trans, 917–927.

Coakley, D., P. Raftery, P. Molloy, and G. White(2011, 14-16 November). Calibration of a detailedbes model to measured data using an evidence-based analytical optimisation approach. In Pro-ceedings of Building Simulation 2011: 12th Confer-ence of International Building Performance Simu-lation Association, Sydney.

Proceedings of the 15th IBPSA ConferenceSan Francisco, CA, USA, Aug. 7-9, 2017

857

Costola, D., J. Clarke, N. Kelly, and F. Monari(2017). A big data approach to the applicationof building performance simulation to improve theoperational performance of large estates. Acceptedfor publication in Proceedings of Building Simu-lation 2017: 15th Conference of the InternationalBuilding Performance Simulation Association, SanFrancisco.

Gelman, A. and D. B. Rubin (1992). Inference fromiterative simulation using multiple sequences. Sta-tistical Science 7 (4), 457–472.

Heo, Y., G. Augenbroe, and R. Choudhary (2013).Quantitative risk management for energy retrofitprojects. Journal of Building Performance Simu-lation 6(4), 257–268.

Heo, Y., R. Choudhary, and G. Augenbroe (2012).Calibration of building energy models for retrofitanalysis under uncertainty. Energy and Build-ings 47, 550–560.

Heo, Y., D. J. Graziano, L. Guzowski, and R. T.Muehleisen (2015). Evaluation of calibration ef-ficacy under different levels of uncertainty. Journalof Building Performance Simulation 8 (3), 135–144.

Higdon, D., J. Gattiker, B. Williams, and M. Right-ley (2008). Computer model calibration using high-dimensional output. Journal of the American Sta-tistical Association 103(482), 570–583.

Johnson, N. and H. Hu (2012). Developing an ef-fective and reliable approach for building energyaudit in support of retrofit decision making. InInternational High Performance Building Confer-ence, Purdue.

Jolliffe, I. (2002). Principal Component Analysis(Springer Series in Statistics). Springer.

Kennedy, M. and A. O’Hagan (2001). Bayesian cali-bration of computer models. Journal of the RoyalStatistical Society Series B 63, 425–464.

Kim, Y., K. Kim, C. Park, and I. Kim (2014, Novem-ber). Deterministic vs. stochastic calibration ofenergy simulation model for an existing building.In ASim 2014, IBPSA Asia Conference, NasogaJapan, 2014.

Lamboni, M., H. Monod, and D. Makowski (2011,Apr). Multivariate sensitivity analysis to mea-sure global contribution of input factors in dy-namic models. Reliability Engineering & SystemSafety 96 (4), 450459.

Monari, F. (2016). Sensitivity Analysis and BayesianCalibration of Building Energy Models. Ph. D. the-sis, University of Strathclyde, Glasgow, UK.

Nelder, J. A. and R. Mead (1965, January). A simplexmethod for function minimization. The ComputerJournal 7 (4), 308–313.

Pedrini, A., F. Westphal, and R. Lambertsb (2002).A methodology for building energy modelling andcalibration in warm climates. Building and Envi-ronment 37, 903–912.

Plummer, M., N. Best, K. Cowles, and K. Vines(2006). CODA: Convergence Diagnosis and Out-put Analysis for MCMC. R News 6 (1), 7–11.

Raftery, P., M. Keane, and A. Costa (2011a). Cal-ibrating whole building energy models: Detailedcase study using hourly measured data. Energyand Buildings 43 (12), 3666 – 3679.

Raftery, P., M. Keane, and J. O’Donnell (2011b).Calibrating whole building energy models: Anevidence-based methodology. Energy and Build-ings 43 (9), 2356 – 2364.

Rasmussen, C. and C. Williams (2006). GaussianProcess for Machine Learning. the MIT Press.

Ratto, M. and A. Pagano (2010, Dec). Using recursivealgorithms for the efficient identification of smooth-ing spline ANOVA models. AStA Advances in Sta-tistical Analysis 94 (4), 367388.

Reddy, T., I. Maor, and C. Panjapornpon (2007a).Calibrating detailed building energy simulationprograms with measured data–part ii: applica-tion to three case study office buildings (rp-1051).HVAC & R Research 13 (2), 221–241.

Reddy, T., I. Maor, I., and C. Panjapornpon(2007b). Calibrating detailed building energy sim-ulation programs with measured data–part i: gen-eral methodology (rp-1051). HVAC & R Re-search 13 (2), 243–265.

Rosenthal, J. S. (2007, Aug). AMcmc: An r interfacefor adaptive MCMC. Computational Statistics &Data Analysis 51 (12), 5467–5470.

Sun, J. and T. Reddy (2006). Calibration of build-ing energy simulation programs using the analyticoptimization approach (rp-1051). HVAC&R RE-SEARCH 12(1), 177–196.

Westphal, F. S. and R. Lamberts (2005). Buildingsimulation calibration using sensitivity analysis. InNinth International IBPSA Conference Montral,Canada August 15-18.

Yoon, J., E. Lee, and D. Claridge (2003). Calibrationprocedure of energy performance simulation modelfor a commercial building. Journal of Solar EnergyEngineering 125(August), 251–257.