Summarizing Measured Data Andy Wang CIS 5930-03 Computer Systems Performance Analysis.
Selecting Evaluation Techniques Andy Wang CIS 5930 Computer Systems Performance Analysis.
-
Upload
cory-wells -
Category
Documents
-
view
219 -
download
0
Transcript of Selecting Evaluation Techniques Andy Wang CIS 5930 Computer Systems Performance Analysis.
![Page 1: Selecting Evaluation Techniques Andy Wang CIS 5930 Computer Systems Performance Analysis.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649e925503460f94b97fc7/html5/thumbnails/1.jpg)
Selecting Evaluation Techniques
Andy WangCIS 5930
Computer SystemsPerformance Analysis
![Page 2: Selecting Evaluation Techniques Andy Wang CIS 5930 Computer Systems Performance Analysis.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649e925503460f94b97fc7/html5/thumbnails/2.jpg)
Decisions to be Made
• Evaluation Technique• Performance Metrics• Performance Requirements
![Page 3: Selecting Evaluation Techniques Andy Wang CIS 5930 Computer Systems Performance Analysis.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649e925503460f94b97fc7/html5/thumbnails/3.jpg)
Evaluation Techniques
Experimentation isn’t always the answer.
Alternatives:• Analytic modeling (queueing theory)• Simulation• Experimental measurementBut always verify your conclusions!
![Page 4: Selecting Evaluation Techniques Andy Wang CIS 5930 Computer Systems Performance Analysis.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649e925503460f94b97fc7/html5/thumbnails/4.jpg)
Analytic Modeling
• Cheap and quick• Don’t need working system• Usually must simplify and make
assumptions
![Page 5: Selecting Evaluation Techniques Andy Wang CIS 5930 Computer Systems Performance Analysis.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649e925503460f94b97fc7/html5/thumbnails/5.jpg)
Simulation
• Arbitrary level of detail• Intermediate in cost, effort, accuracy• Can get bogged down in model-building
![Page 6: Selecting Evaluation Techniques Andy Wang CIS 5930 Computer Systems Performance Analysis.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649e925503460f94b97fc7/html5/thumbnails/6.jpg)
Measurement
• Expensive• Time-consuming• Difficult to get detail• But accurate
![Page 7: Selecting Evaluation Techniques Andy Wang CIS 5930 Computer Systems Performance Analysis.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649e925503460f94b97fc7/html5/thumbnails/7.jpg)
SelectingPerformance
Metrics• Three major perfomance metrics:
– Time (responsiveness)– Processing rate (productivity)– Resource consumption (utilization)
• Error (reliability) metrics:– Availability (% time up)– Mean Time to Failure (MTTF/MTBF)
• Same as mean uptime
• Cost/performance
![Page 8: Selecting Evaluation Techniques Andy Wang CIS 5930 Computer Systems Performance Analysis.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649e925503460f94b97fc7/html5/thumbnails/8.jpg)
Response Time
• How quickly does system produce results?
• Critical for applications such as:– Time sharing/interactive systems– Real-time systems– Parallel computing
![Page 9: Selecting Evaluation Techniques Andy Wang CIS 5930 Computer Systems Performance Analysis.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649e925503460f94b97fc7/html5/thumbnails/9.jpg)
Examples of Response Time
• Time from keystroke to echo on screen• End-to-end packet delay in networks
– How about dropped packets?
• OS bootstrap time• Leaving Love to getting food in Oglesby
– Edibility is not a factor
![Page 10: Selecting Evaluation Techniques Andy Wang CIS 5930 Computer Systems Performance Analysis.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649e925503460f94b97fc7/html5/thumbnails/10.jpg)
Measures of Response Time
• Response time: request-response interval– Measured from end of request– Ambiguous: beginning or end of response?
• Reaction time: end of request to start of processing
• Turnaround time: start of request to end of response
![Page 11: Selecting Evaluation Techniques Andy Wang CIS 5930 Computer Systems Performance Analysis.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649e925503460f94b97fc7/html5/thumbnails/11.jpg)
The Stretch Factor
• Response time usually goes up with load
• Stretch Factor measures this:
0
2
4
6
1 2 3 4Load
Response Time
Low stretch
High stretch
![Page 12: Selecting Evaluation Techniques Andy Wang CIS 5930 Computer Systems Performance Analysis.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649e925503460f94b97fc7/html5/thumbnails/12.jpg)
Processing Rate
• How much work is done per unit time?• Important for:
– Sizing multi-user systems– Comparing alternative configurations– Multimedia
![Page 13: Selecting Evaluation Techniques Andy Wang CIS 5930 Computer Systems Performance Analysis.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649e925503460f94b97fc7/html5/thumbnails/13.jpg)
Examplesof Processing Rate
• Bank transactions per hour• File-transfer bandwidth• Aircraft control updates per second• Jurassic Park customers per day
![Page 14: Selecting Evaluation Techniques Andy Wang CIS 5930 Computer Systems Performance Analysis.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649e925503460f94b97fc7/html5/thumbnails/14.jpg)
Power Consumption Metric
• Watt (power) = joule (energy) / second• Joule = execution time x watt
(consumed at a given point)– Insufficient to just reduce peak power by 2x
at the cost of increasing the computational time by > 2x
• A device may be powered in spikes
![Page 15: Selecting Evaluation Techniques Andy Wang CIS 5930 Computer Systems Performance Analysis.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649e925503460f94b97fc7/html5/thumbnails/15.jpg)
Network Signal Metric
• dBm = 10 x log(1000 * watt)• 1 watt = 30 dBm
![Page 16: Selecting Evaluation Techniques Andy Wang CIS 5930 Computer Systems Performance Analysis.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649e925503460f94b97fc7/html5/thumbnails/16.jpg)
Measuresof Processing Rate
• Throughput: requests per unit time: MIPS, MFLOPS, MB/s, TPS
• Nominal capacity: theoretical maximum: bandwidth
• Knee capacity: where things go bad• Usable capacity: where response time
hits a specified limit• Efficiency: ratio of usable to nominal
capacity
![Page 17: Selecting Evaluation Techniques Andy Wang CIS 5930 Computer Systems Performance Analysis.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649e925503460f94b97fc7/html5/thumbnails/17.jpg)
Nominal, Knee, andUsable Capacities
Response-Time Limit
Knee
UsableCapacity
Knee Capacity
NominalCapacity
![Page 18: Selecting Evaluation Techniques Andy Wang CIS 5930 Computer Systems Performance Analysis.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649e925503460f94b97fc7/html5/thumbnails/18.jpg)
Resource Consumption
• How much does the work cost?• Used in:
– Capacity planning– Identifying bottlenecks
• Also helps to identify “next” bottleneck
![Page 19: Selecting Evaluation Techniques Andy Wang CIS 5930 Computer Systems Performance Analysis.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649e925503460f94b97fc7/html5/thumbnails/19.jpg)
Examplesof Resource
Consumption• CPU non-idle time• Memory usage• Fraction of network bandwidth needed• Square feet of beach occupied
![Page 20: Selecting Evaluation Techniques Andy Wang CIS 5930 Computer Systems Performance Analysis.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649e925503460f94b97fc7/html5/thumbnails/20.jpg)
Measuresof Resource
Consumption• Utilization:
where u(t) is instantaneous resource usage– Useful for memory, disk, etc.– May be tricky
• Busy disk != doing useful work– Due to mechanical overheads
• If u(t) is always either 1 or 0, reduces to busy time or its inverse, idle time– Useful for network, CPU, etc.
t
dttu0
)(
![Page 21: Selecting Evaluation Techniques Andy Wang CIS 5930 Computer Systems Performance Analysis.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649e925503460f94b97fc7/html5/thumbnails/21.jpg)
Error Metrics
• Successful service (speed)– (Not usually reported as error)
• Incorrect service (reliability)• No service (availability)
![Page 22: Selecting Evaluation Techniques Andy Wang CIS 5930 Computer Systems Performance Analysis.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649e925503460f94b97fc7/html5/thumbnails/22.jpg)
Examples of Error Metrics
• Time to get answer from Google• Dropped Internet packets• ATM down time• Wrong answers from IRS
![Page 23: Selecting Evaluation Techniques Andy Wang CIS 5930 Computer Systems Performance Analysis.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649e925503460f94b97fc7/html5/thumbnails/23.jpg)
Measures of Errors
• Reliability: P(error) or Mean Time Between Errors (MTBE)
• Availability:– Downtime: Time when system is
unavailable, may be measured as Mean Time to Repair (MTTR)
– Uptime: Inverse of downtime, often given as Mean Time Between Failures (MTBF/MTTF)
![Page 24: Selecting Evaluation Techniques Andy Wang CIS 5930 Computer Systems Performance Analysis.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649e925503460f94b97fc7/html5/thumbnails/24.jpg)
Financial Measures
• When buying or specifying, cost/performance ratio is often useful
• Performance chosen should be most important for application
![Page 25: Selecting Evaluation Techniques Andy Wang CIS 5930 Computer Systems Performance Analysis.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649e925503460f94b97fc7/html5/thumbnails/25.jpg)
Characterizing Metrics
• Usually necessary to summarize• Sometimes means are enough• Variability is usually critical
– A mean I-10 freeway speed of 55 MPH doesn’t help plan rush-hour trips
![Page 26: Selecting Evaluation Techniques Andy Wang CIS 5930 Computer Systems Performance Analysis.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649e925503460f94b97fc7/html5/thumbnails/26.jpg)
Types of Metrics
• Global across all users• Individual
First helps financial decisions, second measures satisfaction and cost of adding users
![Page 27: Selecting Evaluation Techniques Andy Wang CIS 5930 Computer Systems Performance Analysis.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649e925503460f94b97fc7/html5/thumbnails/27.jpg)
Choosing What to Measure
Pick metrics based on:• Completeness• (Non-)redundancy• Variability
![Page 28: Selecting Evaluation Techniques Andy Wang CIS 5930 Computer Systems Performance Analysis.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649e925503460f94b97fc7/html5/thumbnails/28.jpg)
Completeness
• Must cover everything relevant to problem– Don’t want awkward questions from boss
or at conferences!
• Difficult to guess everything a priori– Often have to add things later
![Page 29: Selecting Evaluation Techniques Andy Wang CIS 5930 Computer Systems Performance Analysis.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649e925503460f94b97fc7/html5/thumbnails/29.jpg)
Redundancy
• Some factors are functions of others• Measurements are expensive• Look for minimal set• Again, often an interactive process
![Page 30: Selecting Evaluation Techniques Andy Wang CIS 5930 Computer Systems Performance Analysis.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649e925503460f94b97fc7/html5/thumbnails/30.jpg)
Variability
• Large variance in a measurement makes decisions impossible
• Repeated experiments can reduce variance– Expensive– Can only reduce it by a certain amount
• Better to choose low-variance measures to start with
![Page 31: Selecting Evaluation Techniques Andy Wang CIS 5930 Computer Systems Performance Analysis.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649e925503460f94b97fc7/html5/thumbnails/31.jpg)
Classes of Metrics: HB
• Higher is Better:
Utility
Throughput
Better
![Page 32: Selecting Evaluation Techniques Andy Wang CIS 5930 Computer Systems Performance Analysis.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649e925503460f94b97fc7/html5/thumbnails/32.jpg)
Classes of Metrics: LB
• Lower is Better:
Utility
Response Time
Better
![Page 33: Selecting Evaluation Techniques Andy Wang CIS 5930 Computer Systems Performance Analysis.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649e925503460f94b97fc7/html5/thumbnails/33.jpg)
Classes of Metrics: NB
• Nominal is Best:
Free Disk Space
Best
Utility
![Page 34: Selecting Evaluation Techniques Andy Wang CIS 5930 Computer Systems Performance Analysis.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649e925503460f94b97fc7/html5/thumbnails/34.jpg)
Setting Performance Requirements
Good requirements must be SMART:• Specific• Measurable• Acceptable• Realizable• Thorough
![Page 35: Selecting Evaluation Techniques Andy Wang CIS 5930 Computer Systems Performance Analysis.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649e925503460f94b97fc7/html5/thumbnails/35.jpg)
Example: Web Server
• Users care about response time (end of response)
• Network capacity is expensive want high utilization
• Pages delivered per day matters to advertisers
• Also care about error rate (failed & dropped connections)
![Page 36: Selecting Evaluation Techniques Andy Wang CIS 5930 Computer Systems Performance Analysis.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649e925503460f94b97fc7/html5/thumbnails/36.jpg)
Example: Requirementsfor Web Server
• 2 seconds from request to first byte, 5 to last
• Handle 25 simultaneous connections, delivering 100 KB/s to each
• 60% mean utilization, with 95% or higher less than 5% of the time
• <1% of connection attempts rejected or dropped
![Page 37: Selecting Evaluation Techniques Andy Wang CIS 5930 Computer Systems Performance Analysis.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649e925503460f94b97fc7/html5/thumbnails/37.jpg)
Is the Web Server SMART?
• Specific: yes• Measurable: may have trouble with
rejected connections• Acceptable: response time and aggregate
bandwidth might not be enough• Realizable: requires T3 link; utilization
depends on popularity• Thorough? You decide
![Page 38: Selecting Evaluation Techniques Andy Wang CIS 5930 Computer Systems Performance Analysis.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649e925503460f94b97fc7/html5/thumbnails/38.jpg)
RemainingWeb Server Issues
• Redundancy: response time is closely related to bandwidth, utilization
• Variability: all measures could vary widely.– Should we specify variability limits for other
than utilization?
![Page 39: Selecting Evaluation Techniques Andy Wang CIS 5930 Computer Systems Performance Analysis.](https://reader030.fdocuments.us/reader030/viewer/2022032709/56649e925503460f94b97fc7/html5/thumbnails/39.jpg)
White Slide