Test-Driven Simulation Modelling (TDSM)

Bhakti Stephan Onggo Department of Management Science s.onggo@lancaster.ac.uk

• We propose a novel technique for the verification and validation of simulation models

• The technique explicitly embeds verification and validation to simulation modelling and it can be implemented using standard unit testing tools

• The proposed technique has been tested on ABS (Agent-Based Simulation) and DES (Discrete-Event Simulation)

Summary

2

• Model verification and validation is an important activity in simulation • Verification: to check the correctness of a simulation model

against its conceptual model • Validation: to build confidence that the model is adequate for its

intended use • In an incremental model development, we need to make sure a new

addition to the model does not break the correctness and validity of the earlier version of the model • Iterative -> automate the verification and validation?

• Model documentation is a good practice, so does documentation about how to verify and validate the model • Accompany a model with its verification and validation cases

• It is especially important if a significant part of the model is written in a programming language (e.g. Visual Logic, Java)

Why is our research important?

3

• Hence, the contributions of our work are • Proposing a method to help with simulation model verification

and validation that can be automated using a software tool such as a unit testing tool

• Our method promotes the principle of parsimony (incremental model development)

• Our method encourages modellers to think about V&V early • Our method promotes documenting verification and validation

cases of a simulation model

Why is our research important?

4

• Test-Driven Development (TDD) in software engineering

• TDD is suitable for a software development process that is iterative, incremental and evolutionary (such as simulation)

• Exploratory study on the use of TDD as a simulation model verification tool has been done (Collier & Ozik, Asta et al. and Onggo et al.)

• Similar approach to TDD has been explored too (Gurcan et al., Niazzi et al. and Thiele et al.)

Asta, S., E. Özcan, and P.-O. Siebers. 2014. Collier, N., and J. Ozik. 2013 Onggo, B.S.S., C. Indriany, and M. Gunal. 2014 Gurcan, O., O. Dikenelli, C. Bernon. 2013. Niazzi M.A., A. Hussain, and M. Kolberg. 2009 Thiele, J.C., Kurth, W., and Grimm V. 2014.

Background

5

Method • We extended TDD to cover model

validation and embed TDD into the standard simulation modelling process – we call this Test-Driven Simulation Modelling (TDSM)

• For each verification or validation case, we need to specify how the model (or part of the model) should behave

• A validation case may require the model to be run multiple times to build the confidence intervals

• Sensitivity analysis can be specified in which a validation case will test one point in the exploration space

6

Case 1: DES • SharpSim - a general

purpose DES library written in Visual CSharp (C#)

• M/M/1 Queue

RUN ARRIVAL STARTSERVICE

ENDSERVICE

ID = 1;S = 2; Customer c = new Customer(ID);edgeR_A.attribute = c;

eStart.queue.Add (ev.evnt.parameter);edgeA_S.attribute = eStart.queue[0]; ID++;Customer c = new Customer(ID);edgeA_A.attribute = c;if (S > 0) edgeA_S.condition = true;else edgeA_S.condition = false;

(S > 0)

∫

∫(Q > 0)

eStart.queue .RemoveAt(0); S--;edgeS_E.condition = true;edgeS_E.attribute = c;

S++; if (eStart.queue .Count() == 0) edgeE_S.condition = false; else { edgeE_S.condition = true; edgeE_S.attribute = eStart.queue[0]; } Stats.Collect

tsta

ta

7

Case 1: DES Verification

Scenario 1: check correctness if arrivals at 1, 2 and service time 4, 3

8

Case 1: DES Validation Scenario 2: Validate the model by evaluating whether the expected waiting time from the analytic model (i.e. 0.1922) is within the 95% confidence interval of the expected waiting time from the simulation model

9

Case 2: ABS • Agent-based maritime

search-operation simulation (MASSIM) – Onggo & Karatas (Wintersim 2014)

• Various search algorithms: random, exhaustive, parallel, creeping, etc.

• Detection functions: deterministic, various probabilistic functions

• Single searcher, multiple searchers

10

Case 2: ABS Validation

Exhaustive search

Random search

11

Case 2: ABS Validation Parallel search

12

• TDSM is a useful technique in the verification and validation of simulation models

• TDSM works based on two principles: • modellers should be guided by the knowledge of relevant

analytic models that are simpler and have known solutions when they are building a complex model

• Modellers must think about how their model is going to be verified and validated, even before they begin to develop the model

• TDSM and other methods (Gurcan et al. 2013, Niazzi et al. 2009 and Thiele et al. 2014) aim to help modellers validate a simulation model

• TDSM is useful for auto-verification or auto-validation of simulation models

Conclusion

13

• Evaluate TDSM using an experiment with real modellers • Find an optimum TDSM strategy that balances test coverage and

cost • Model verification and validation is a combinatorial problem

• We need tool to evaluate the test cases (unit tests) • A unit test can contain mistakes too!

• We need to embed TDSM components in a Visual Simulation Modelling tool

• Get some feedback

Next steps/future work

14

References

• Onggo BSS and Karatas M. (2016) 'Test-driven simulation modelling: A case study using agent-based maritime search-operation simulation', European Journal of Operational Research, 254 (2):517-531

• Onggo, B. S. S. , Indriany, C. & Gunal, M. (2014). Test-driven simulation modelling. In A. Bruzzone, E. Çayırcı, M. Günal, A. R. Kaylan, M. Massei, & C. Zanni-Merk (Eds.), Proceedings of the 7th international workshop on applied modeling and sim- ulation (WAMS14) (pp. 43–48). Turkey: Istanbul .

15