CPSC 531: Experiment Design1 CPSC 531: Experiment Design and Performance Evaluation Instructor:...

CPSC 531: Experiment Design 1

CPSC 531: Experiment Design and Performance Evaluation

Instructor: Anirban MahantiOffice: ICT 745Email: [email protected] Location: TRB 101Lectures: TR 15:30 – 16:45 hoursClass web page:

http://pages.cpsc.ucalgary.ca/~mahanti/teaching/F05/CPSC531

Slides courtesy Professor Carey Williamson (minor modifications by Anirban Mahanti).




PERFORMANCE EVALUATION

Often one needs to design and conduct an experiment in order to: demonstrate that a new technique or

concept is feasibledemonstrate that a new method is

better than an existing methodunderstand the impact of various

factors and parameters on the overall system performance


PERFORMANCE EVALUATION There is a whole field of computer science

called computer systems performance evaluation that does exactly this

One of the best books is Raj Jain’s “The Art of Computer Systems Performance Analysis”, Wiley & Sons, 1991 (listed in bibliography)

Much of what is outlined in this presentation is described in more detail in [Jain 1991]


PERF EVAL 101: THE BASICS There are three main methods used in

the design of performance studies Experimental approaches

measurement and use of a real system Analytic approaches

the use of mathematics, queueing theory, Petri Nets, abstract models, etc

Simulation approaches design and use of computer simulations and

simplified models to assess performance


EXPERIMENTAL DESIGN AND METHODOLOGY

The design of a performance study requires great care in experimental design and methodology

Need to identify experimental factors to be tested levels (settings) for these factors performance metrics to be used experimental design to be used


FACTORS Factors are the main “components” or

“things” that are to be varied in an experiment, because their impact on performance wants to be understood

Examples: switch size, network load, number of buffers at output ports

Need to choose factors properly, since the number of factors affects size of study


LEVELS Levels are the precise settings of the

factors that are to be used in an experiment

Examples: switch size N = 2, 4, 8, 16 Example: buffer size B = 100, 200, 400,

800 Need to choose levels realistically Need to cover reasonable portion of the

design space


PERFORMANCE METRICS Performance metrics specify what you want

to measure in your performance study Examples: packet loss, packet delay Must choose your metrics properly and

instrument your experiment accordingly


EXPERIMENTAL DESIGN Experimental design refers to the

organizational structure of your experiment Need to methodically go through factors

and levels to get the full range of experimental results desired

There are several “classical” approaches to experimental design


Experiment Design - Classification

One factor at a time vary only one factor through its levels to see

what the impact is on performance Two factors at a time

vary two factors to see not only their individual effects, but also their interaction effects, if any

Full factorial try every possible combination of factors

and levels to see full range of performance results


Example: Web Proxy Caching

Web Clients

Web Servers

Web Proxy Server

…….

…….

Filtering

Input stream

Filtered stream (misses)


Example: Factors and Levels

Cache size Cache Replacement Policy

Recency-based LRU Frequency-based LFU-Aging Size-based GD-Size RAND FIFO

Workload Characteristics One-timers, Zipf slope, tail index, correlation,

temporal locality model


Example: Performance Metrics Document Hit Ratio (DHR)

What percentage of the documents requested by the clients are satisfied directly by the proxy, without having to obtain from the Web server?

Byte Hit Ratio (BHR) What percentage of the bytes requested by the

clients are satisfied directly by the proxy, without having to obtain from the Web server?


Example: Performance Results

Workload: Weak temporal locality Workload: Strong temporal locality

Document Hit Ratio


OTHER ISSUES Simulation run length

choosing a long enough run time to get statistically meaningful results (equilibrium)

Simulation start-up effects and end effects deciding how much to “chop off” at the start

and end of simulations to get proper results Replications

ensure repeatability of results, and gain greater statistical confidence in the results given

Presentation of results


Presentation of Results

Graphs and tables are the two most common ways of illustrating and/or summarizing data graphs can show you the trends tables provide the details

There are good ways and bad ways to do each of these

Again, it is a bit of an “art”, but there are lots of good tips and guidelines as well


Table Tips Decide if a table is really needed; if so, should

it be part of main report, or just an appendix? Choose formatting software with which you are

familiar; easy to import data, export tables Table caption goes at the top Clearly delineate rows and columns (lines) Logically organize rows and columns Report results to several significant digits Be consistent in formatting wherever possible


Graphing Tips Choose a good software package,

preferably one with which you are familiar, and one for which it is easy to import data, export graphs

Title at top; caption below (informative) Labels on each axis, including units Logical step sizes along axes (10’s,

100’s…) Make sure choice of scale is clear for each

axis (linear, log-linear, log-log) Graph should start from origin (zero) unless

there is a compelling reason not to do so


Graphing Tips (cont’d) Make judicious choice of type of plot

scatter plot, line graph, bar chart, histogram Make judicious choice of line types

solid, dashed, dotted, lines and points, colours If multiple lines on a plot, then use a key,

which should be well-placed and informative If graph is “well-behaved”, then organize the

key to match the lines on the graph (try it!) Be consistent from one graph to the next

wherever possible (size, scale, key, colours)


SUMMARY Great care must be taken in

experimental design and methodology if the experiment is to achieve its goal, and if results are to be fully understood

Computer systems performance evaluation defines standard methods for designing and conducting performance studies

Please follow these guidelines (where applicable) when doing assignments and course projects

CPSC 531: Experiment Design1 CPSC 531: Experiment Design and Performance Evaluation Instructor:...

Documents

Transcript of CPSC 531: Experiment Design1 CPSC 531: Experiment Design and Performance Evaluation Instructor:...