Http://dblab.usc.edu Announcements URL for the class is URL for the class is .

http://dblab.usc.eduhttp://dblab.usc.edu

AnnouncementsAnnouncements URL for the class is URL for the class is

http://dblab.usc.edu/csci599http://dblab.usc.edu/csci599 I will be away from September 5 to 16. There I will be away from September 5 to 16. There

will be lectures: will be lectures: Video-taped presentation on automatic system Video-taped presentation on automatic system

tuning.tuning. Use these presentations to complete homework Use these presentations to complete homework

1 (posted on the web site).1 (posted on the web site). Design your project.Design your project.

Prof. Roger Zimmermann is our Guest Prof. Roger Zimmermann is our Guest Lecturer on September 28Lecturer on September 28thth..

Today’s paper: Today’s paper: T. R. Andel and A. Yasinac. On the Credibility of T. R. Andel and A. Yasinac. On the Credibility of

Manet Simulations. IEEE Computer, July 2006.Manet Simulations. IEEE Computer, July 2006.

http://dblab.usc.edu/csci599

10 Commandments of 10 Commandments of Simulation StudiesSimulation Studies

Shahram GhandeharizadehShahram GhandeharizadehComputer Science DepartmentComputer Science Department

University of Southern CaliforniaUniversity of Southern California


OutlineOutline Why simulation studies?Why simulation studies? 10 Commandments of simulation studies.10 Commandments of simulation studies. Review of each commandment.Review of each commandment. ConclusionsConclusions ReferencesReferences


Why simulation models?Why simulation models? To understand tradeoffs between alternative To understand tradeoffs between alternative

ways of implementing a functionality when:ways of implementing a functionality when: A realistic implementation is too expensive/time-A realistic implementation is too expensive/time-

consuming.consuming.

Quantify the complexity of alternative Quantify the complexity of alternative implementations prior to a real system implementations prior to a real system prototyping effort. Make sure it is not over-prototyping effort. Make sure it is not over-engineered: Many fold increase in software engineered: Many fold increase in software to get a 2% benefit?to get a 2% benefit?

Quantify tradeoffs with alternative strategies Quantify tradeoffs with alternative strategies and choose the most appropriate one.and choose the most appropriate one.

Gain insight to develop better strategies.Gain insight to develop better strategies.

Byte-hit Byte-hit versus versus FrequencFrequency-based.y-based.

Identical Identical Complexity.Complexity.

Byte-hit outperforms Frequency-Byte-hit outperforms Frequency-based with larger configurations.based with larger configurations.

Halo-clip and Halo-clip and Halo-block.Halo-block.


Why simulation models?Why simulation models? There are other reasons for developing a There are other reasons for developing a

simulation model that falls beyond the scope simulation model that falls beyond the scope of this presentation:of this presentation: To describe and understand systems too To describe and understand systems too

complex to explain, e.g., the human brain.complex to explain, e.g., the human brain. To identify factors/conditions with most impact To identify factors/conditions with most impact

on a performance metric, e.g., airplane crash in on a performance metric, e.g., airplane crash in Singapore during a hurricane. Typically Singapore during a hurricane. Typically conducted using traces gathered from a real conducted using traces gathered from a real environment.environment.

Capacity planning and risk analysis, e.g., design Capacity planning and risk analysis, e.g., design of power grids in anticipation of 20% increase in of power grids in anticipation of 20% increase in peak load in 2007, war-games.peak load in 2007, war-games.


Inherent limitationInherent limitation A simulation is abstraction of a real system A simulation is abstraction of a real system

that may either exist or foreseen to exist in that may either exist or foreseen to exist in the future.the future.

How do you know the abstraction level is How do you know the abstraction level is correct?correct?


Inherent limitationInherent limitation A simulation is abstraction of a real system A simulation is abstraction of a real system

that may either exist or foreseen to exist in that may either exist or foreseen to exist in the future.the future.

How do you know the abstraction level is How do you know the abstraction level is correct?correct?1.1. We cannot know what we do not know.We cannot know what we do not know.

2.2. We can never be sure that we have accounted for We can never be sure that we have accounted for all aspects that could affect a simulation model’s all aspects that could affect a simulation model’s ability to provide meaningful results.ability to provide meaningful results.

3.3. Items 1 and 2 are specially true for future Items 1 and 2 are specially true for future foreseen applications that do not exist today.foreseen applications that do not exist today.


Alternatives to a simulationAlternatives to a simulation

Real Prototypes/ImplementationReal Prototypes/Implementation

Simulation StudySimulation Study

Analytical ModelsAnalytical Models






An implementation is specific and most often An implementation is specific and most often static. A simulation study can model a variety of static. A simulation study can model a variety of possible system parameters.possible system parameters.

More AbstractMore Abstract

More SpecificMore Specific






A real prototype is more expensive to design, A real prototype is more expensive to design, implement, test, and maintain.implement, test, and maintain.

CheaperCheaper

ExpensiveExpensive






Analytical models are more general, providing Analytical models are more general, providing insights that might be overlooked by a narrow insights that might be overlooked by a narrow implementation. implementation.

GeneralGeneral

NarrowNarrow


Components of a SimulatorComponents of a Simulator1.1. Abstraction of a computing environment.Abstraction of a computing environment.

DMS’06: Mass storage devices to store clips, DMS’06: Mass storage devices to store clips, admission control, streaming of data as a admission control, streaming of data as a function of time.function of time.

Manet: Physical and Data link layers.Manet: Physical and Data link layers.

2.2. Implementation of alternative strategies.Implementation of alternative strategies. DMS’06: Byte-hit and Frequency-basedDMS’06: Byte-hit and Frequency-based Manet: Alternative network routing protocols Manet: Alternative network routing protocols

such as DSR and AODV.such as DSR and AODV.

3.3. Abstractions of an application.Abstractions of an application. How requests are issued? Uniformly or in a How requests are issued? Uniformly or in a

bursty manner?bursty manner? What processing is required from the system?What processing is required from the system?


10 Commandments10 Commandments1.1. Thou shall NOT obtain results from a computing Thou shall NOT obtain results from a computing

environment unless its behavior is validated.environment unless its behavior is validated.

2.2. Thou shall NOT obtain results from an incorrect Thou shall NOT obtain results from an incorrect implementation of your strategies.implementation of your strategies.

3.3. Thou shall NOT use unrealistic computing Thou shall NOT use unrealistic computing environments.environments.

4.4. Thou shall NOT use unrealistic workloads or Thou shall NOT use unrealistic workloads or applications.applications.

5.5. Thou shall NOT use someone else’s computing Thou shall NOT use someone else’s computing environment as the basis for your implementation environment as the basis for your implementation unless you understand its abstractions and unless you understand its abstractions and behavior for different parameter settings.behavior for different parameter settings.

6.6. Thou shall NOT focus on absolute values. (Focus on Thou shall NOT focus on absolute values. (Focus on the observed trends.)the observed trends.)


10 Commandments (Cont…)10 Commandments (Cont…)7.7. Thou shall NOT make a big deal out of Thou shall NOT make a big deal out of

differences lower than 15%. differences lower than 15%.

8.8. Thou shall NOT report on a simulation with a Thou shall NOT report on a simulation with a few runs; establish statistical confidence.few runs; establish statistical confidence.

9.9. Thou shall NOT publish results obtained Thou shall NOT publish results obtained from a simulator without disclosing all your from a simulator without disclosing all your assumptions. (Results must be assumptions. (Results must be reproducible.)reproducible.)

10.10. Thou shall NOT perceive simulation as an Thou shall NOT perceive simulation as an end in itself. (Validate your results against a end in itself. (Validate your results against a real implementation.) real implementation.)


1. Do NOT obtain results from a computing 1. Do NOT obtain results from a computing environment unless its behavior is validatedenvironment unless its behavior is validated

Limitation: Trends observed from simple Limitation: Trends observed from simple flooding using Opnet, Network Simulator flooding using Opnet, Network Simulator (NS-2), and Global Mobile Information (NS-2), and Global Mobile Information Systems Simulation Library (GloMoSim) do Systems Simulation Library (GloMoSim) do NOT agree!NOT agree!

Claim:Claim: Do you agree?Do you agree?

What to do? What to do?


1. Do NOT obtain results from a computing 1. Do NOT obtain results from a computing environment unless its behavior is validatedenvironment unless its behavior is validated

With simulation model of Byte-hit & With simulation model of Byte-hit & Frequency-based, there are sanity checks to Frequency-based, there are sanity checks to verify correctness of:verify correctness of: Simulating time,Simulating time, Usage of network bandwidth,Usage of network bandwidth, Reservation of bandwidth,Reservation of bandwidth, Generating requests based on a Zipfian Generating requests based on a Zipfian

distribution,distribution, Maintaining information about assignment of Maintaining information about assignment of

clips to nodes.clips to nodes.

A few of these sanity checks are published A few of these sanity checks are published at:at: http://dblab.usc.edu/Users/Shahram/DownloadPage.htmhttp://dblab.usc.edu/Users/Shahram/DownloadPage.htm


2. Do NOT obtain results from an incorrect 2. Do NOT obtain results from an incorrect implementation of your strategies.implementation of your strategies.

Make sure your simulation implements desired functionality, Make sure your simulation implements desired functionality, e.g.,e.g.,


2. Do NOT obtain results from an incorrect 2. Do NOT obtain results from an incorrect implementation of your strategies (Cont…)implementation of your strategies (Cont…)

A strategy is realized at three different abstractions:A strategy is realized at three different abstractions: ConceptualConceptual

This is a description of the strategy at a very abstract level. It is This is a description of the strategy at a very abstract level. It is written in a paper.written in a paper.

LogicalLogical This is a description of the strategy at a pseudo-code level. It This is a description of the strategy at a pseudo-code level. It

contains data structures and one may perform complexity analysis on contains data structures and one may perform complexity analysis on the strategy.the strategy.

PhysicalPhysical This is implementation of the strategy in a simulator (or a real This is implementation of the strategy in a simulator (or a real

system). It consists of actual code that one debugs and executes in a system). It consists of actual code that one debugs and executes in a computing environment.computing environment.

This Testament applies to all 3 abstractions.This Testament applies to all 3 abstractions.


2. Do NOT obtain results from an incorrect 2. Do NOT obtain results from an incorrect implementation of your strategies (Solutions).implementation of your strategies (Solutions).

Conceptual:Conceptual: Ask colleagues to review your conceptual description of a strategy.Ask colleagues to review your conceptual description of a strategy. Do a literature search to see if others have considered strategies similar to Do a literature search to see if others have considered strategies similar to

yours in other disciplines.yours in other disciplines. Document the limitations of your strategy.Document the limitations of your strategy. Be aware of your assumptions and document them.Be aware of your assumptions and document them.

Physical:Physical: Check your implementation at every opportunity by maintaining your sanity Check your implementation at every opportunity by maintaining your sanity

tests as an integral component of the simulator.tests as an integral component of the simulator. If the results from a run look strange, be able to run your sanity tests immediately.If the results from a run look strange, be able to run your sanity tests immediately.

Try to implement your strategies on a small scale using a prototype.Try to implement your strategies on a small scale using a prototype. Validate simulation models against real-world implementation and environments.Validate simulation models against real-world implementation and environments.

Examine your parameter settings and ask are they real, goes to Testaments 3 Examine your parameter settings and ask are they real, goes to Testaments 3 and 4.and 4. Be objective and if you have idle computing resource, setup a schedule to run all the Be objective and if you have idle computing resource, setup a schedule to run all the

possible parameter settings to VERIFY correctness of your strategy. Start with the possible parameter settings to VERIFY correctness of your strategy. Start with the extreme settings and their interactions. Based on the obtained insights, narrow the extreme settings and their interactions. Based on the obtained insights, narrow the possibilities down to those that you speculate to be interesting. Verify the interesting possibilities down to those that you speculate to be interesting. Verify the interesting experiments. Avoid surprised by verifying those experiments that you consider not experiments. Avoid surprised by verifying those experiments that you consider not interesting. For each, examine the behavior of your strategy and be able to explain it. interesting. For each, examine the behavior of your strategy and be able to explain it. If you canNOT explain the behavior of your strategy then something is wrong. One If you canNOT explain the behavior of your strategy then something is wrong. One possibility is that your proposed strategy is not implemented correctly.possibility is that your proposed strategy is not implemented correctly.


3. Do NOT use unrealistic computing environments.3. Do NOT use unrealistic computing environments.


4. Do NOT use unrealistic workloads or applications.4. Do NOT use unrealistic workloads or applications.

Possible workloads:Possible workloads: Synthetic: Generated using a random number generator.Synthetic: Generated using a random number generator. Trace-driven: More realistic, obtained from an environment.Trace-driven: More realistic, obtained from an environment. Hybrid: Derive a synthetic workload starting with a trace.Hybrid: Derive a synthetic workload starting with a trace.

Are synthetic and hybrid workload real?Are synthetic and hybrid workload real?


4. Do NOT use unrealistic workloads or applications.4. Do NOT use unrealistic workloads or applications.

Possible workloads:Possible workloads: Synthetic: Generated using a random number generator.Synthetic: Generated using a random number generator. Trace-driven: More realistic, obtained from an environment.Trace-driven: More realistic, obtained from an environment. Hybrid: Derive a synthetic workload starting with a trace.Hybrid: Derive a synthetic workload starting with a trace.

Are synthetic and hybrid workload real?Are synthetic and hybrid workload real?

Are trace-driven workloads general purpose enough?Are trace-driven workloads general purpose enough?

Solution: Use a trace-driven workload in combination with a Solution: Use a trace-driven workload in combination with a synthetic/hybrid workload.synthetic/hybrid workload.


5. Do NOT use someone else’s computing environment 5. Do NOT use someone else’s computing environment as the basis for your implementation unless you as the basis for your implementation unless you understand its abstractions and behavior for different understand its abstractions and behavior for different parameter settings.parameter settings.

If you decide to use a network simulator If you decide to use a network simulator such as NS-2, make sure you understand its such as NS-2, make sure you understand its different parameter settings and their impact different parameter settings and their impact on obtained results:on obtained results: Relative ranking between AODV and DSR change Relative ranking between AODV and DSR change

depending on the parameter settings of depending on the parameter settings of GloMoSim.GloMoSim.

Solution: Make sure you understand different Solution: Make sure you understand different parameters of a simulator and how they will parameters of a simulator and how they will impact your observed trends.impact your observed trends.


5. Do NOT use someone else’s computing environment 5. Do NOT use someone else’s computing environment as the basis for your implementation unless you as the basis for your implementation unless you understand its abstractions and behavior for different understand its abstractions and behavior for different parameter settings.parameter settings.

Make sure you understand a software Make sure you understand a software package such as GloMoSim prior to using it:package such as GloMoSim prior to using it: Document all its parameter settings and how the Document all its parameter settings and how the

simulator behaves.simulator behaves. When reading manuals and other’s technical When reading manuals and other’s technical

papers, recreate their experiments to see if you papers, recreate their experiments to see if you can re-produce their results.can re-produce their results.

Run simple experiments to see if they are Run simple experiments to see if they are supported correctly.supported correctly. An experiment is simple when its result is known in An experiment is simple when its result is known in

advance.advance.


6. Do NOT focus on absolute values. (Focus on the 6. Do NOT focus on absolute values. (Focus on the observed trends.)observed trends.)

Limitation:Limitation:


6. Do NOT focus on absolute values. (Focus must be on 6. Do NOT focus on absolute values. (Focus must be on the observed trends.)the observed trends.)

Solution:Solution: Describe behavior of protocol A and B for Describe behavior of protocol A and B for

different parameter settings.different parameter settings. Focus on the observed trends with each Focus on the observed trends with each

protocol.protocol. Identify the strengths and weaknesses of each Identify the strengths and weaknesses of each

protocol.protocol. Be aware of technological trends. Be aware of technological trends.

Consider scenarios where your anticipated trends are Consider scenarios where your anticipated trends are outperformed/underperformed.outperformed/underperformed.

Do not make a big deal out of minor differences Do not make a big deal out of minor differences (less than 15% difference, see Testament 7).(less than 15% difference, see Testament 7).


7. Do NOT make a big deal out of differences lower than 7. Do NOT make a big deal out of differences lower than 15%.15%.

A simulator is an abstraction:A simulator is an abstraction:

Small differences (less than 15%) are Small differences (less than 15%) are typically noise attributed to missing typically noise attributed to missing components, lack of detail.components, lack of detail.


8. Do NOT report on a simulation with a few runs; 8. Do NOT report on a simulation with a few runs; establish statistical confidence.establish statistical confidence.


Solution: Establish statistical confidence.Solution: Establish statistical confidence.


Solution: Establish statistical confidence (Cont…)Solution: Establish statistical confidence (Cont…)


9. Do NOT publish results obtained from a simulator 9. Do NOT publish results obtained from a simulator without disclosing all your assumptions.without disclosing all your assumptions.

Study of 114 peer-reviewed manet research Study of 114 peer-reviewed manet research papers published between 2000 to 2005.papers published between 2000 to 2005.


9. Do NOT publish results obtained from a simulator 9. Do NOT publish results obtained from a simulator without disclosing all your assumptions (Cont…)without disclosing all your assumptions (Cont…)

Typical cause: Lack of space. Conference Typical cause: Lack of space. Conference papers are limited in number of pages. papers are limited in number of pages. Describing the simulator may leave no room Describing the simulator may leave no room for stating all assumptions.for stating all assumptions.

Solution:Solution:1.1. Write the complete paper as a technical report.Write the complete paper as a technical report.

2.2. Summarize the key parameters in the Summarize the key parameters in the conference/journal paper and reference the conference/journal paper and reference the technical paper.technical paper.

3.3. Make sure the technical paper is available on the Make sure the technical paper is available on the web.web.


10. Do NOT perceive simulation as an end in itself.10. Do NOT perceive simulation as an end in itself.

Simulation is an abstraction.Simulation is an abstraction. An implementation highlights many important An implementation highlights many important

details (the devil is in the detail).details (the devil is in the detail). An analytical model provides further verification An analytical model provides further verification

for the observations made using a simulator.for the observations made using a simulator.

Simulation is to quantify tradeoffs Simulation is to quantify tradeoffs associated with alternative design strategies associated with alternative design strategies that should be investigated further.that should be investigated further.


Two Popular simulation modelsTwo Popular simulation models

Closed simulation models:Closed simulation models: A system consisting of N Servers,A system consisting of N Servers, M Terminals,M Terminals, A terminal submits one job. It does not submit A terminal submits one job. It does not submit

another job until the pending job completes.another job until the pending job completes. A terminal may think/sleep between its requests.A terminal may think/sleep between its requests.

Open simulation models:Open simulation models: Arrival rate for requests,Arrival rate for requests, A system with a service time,A system with a service time, Queues form when requests arrive and the Queues form when requests arrive and the

system is busy.system is busy.


Two Popular simulation modelsTwo Popular simulation models

Closed simulation models:Closed simulation models: Service time as a function of the number of: Service time as a function of the number of:

1.1. servers, servers,

2.2. terminals, terminals,

3.3. terminal think time.terminal think time.

Open simulation models:Open simulation models: Service time as a function of request arrival rate.Service time as a function of request arrival rate.


Open simulation modelsOpen simulation models Represented as A/S/n where A is the arrival Represented as A/S/n where A is the arrival

process, S is the service time process and n process, S is the service time process and n is the number of servers. A and S may be:is the number of servers. A and S may be:1.1. M (Markovian), Poisson exponential densityM (Markovian), Poisson exponential density

2.2. D (Deterministic), a fixed valueD (Deterministic), a fixed value

3.3. G (General), an arbitrary probability distribution.G (General), an arbitrary probability distribution.

Popular queuing models:Popular queuing models:1.1. M/M/1: A Poisson process generates both the M/M/1: A Poisson process generates both the

arrival and service times. Key input parameters arrival and service times. Key input parameters are arrival rates and service rates.are arrival rates and service rates.

2.2. M/D/1: A Poisson process generates arrival M/D/1: A Poisson process generates arrival times. Service time is deterministic because it is times. Service time is deterministic because it is a fixed value. Key input parameters are arrival a fixed value. Key input parameters are arrival rates and a fixed service time.rates and a fixed service time.


ReferencesReferences T. R. Andel and A. Yasinac. T. R. Andel and A. Yasinac. On the Credibility of Manet SimulationsOn the Credibility of Manet Simulations. IEEE . IEEE

Computer, July 2006.Computer, July 2006. S. Ghandeharizadeh, T. Helmi, T. Jung, S. Kapadia, and S. Shayandeh. An S. Ghandeharizadeh, T. Helmi, T. Jung, S. Kapadia, and S. Shayandeh. An

Evaluation of Two Policies for Placement of Continuous Media in Multi-hop Evaluation of Two Policies for Placement of Continuous Media in Multi-hop Wireless NetworksWireless Networks. USC Database Laboratory Technical Report Number 2006-03. Shorter . USC Database Laboratory Technical Report Number 2006-03. Shorter version appeared in the Twelfth International Conference on Distributed version appeared in the Twelfth International Conference on Distributed Multimedia Systems (DMS 2006), Grand Canyon, Aug 30-Sept 1, 2006. Multimedia Systems (DMS 2006), Grand Canyon, Aug 30-Sept 1, 2006.

R. Jain. The Art of Computer Systems Performance Analysis, John Wiley R. Jain. The Art of Computer Systems Performance Analysis, John Wiley and Sons, 1991.and Sons, 1991.

Http://dblab.usc.edu Announcements URL for the class is URL for the class is .

Documents

Transcript of Http://dblab.usc.edu Announcements URL for the class is URL for the class is .