Measuring InteractivePerformance with VNCplay

Nickolai Zeldovich, Ramesh Chandra

Stanford University

Measuring performance

• Scientific computing or server workloads:– How long does it take for my program to run?

• Measure total running time

Measuring performance

• Users of interactive applications:– I asked for something, when will I get it?

• But we still measure total runtime– WinBench runs Word, PowerPoint, etc as fast

as possible, and measures total runtime

We are measuring the wrong thing

Case in point

• Use Microsoft PowerPoint for 5 minutes

• Two machines: 2.0 GHz and 300 MHz– Expect to see different performance!

• Two performance metrics:– Total runtime: from start to finish– Response time: from click to output on screen

Performance Metrics

0

100

200

300

400

To

tal

Ru

nti

me

(sec

on

ds)

300 MHz 2.0 GHz0

500

1000

1500

2000

Med

ian

Res

po

nse

Tim

e (m

sec)

300 MHz 2.0 GHz

Total Runtime Response Time

Users pause a lot

. . .

0 sec 0.1 sec 0.2 sec 5 sec. . .

Response time is noise in total runtime

Outline

• Overview of VNCplay

• Current approaches

• How VNCplay works

• Evaluation– Quantitative results– Qualitative experience

• Summary

Overview of VNCplay

• To compare interactive performance, we need reproducible workloads– Record interactive user session– Replay session in different environments

• Compare response time between replays

Current approaches

Display protocols: X11, VNC, Windows

Key strokes, mouse clicks

Screen updates

Current approaches

Screen updates

. . .0 sec 1 sec 5 sec 8 secTime

InputEvent

Recorder

Tools like Xnee, Visual Test, AutoIt, rfbplaymacro

Current approaches

Screen updates

. . .0 sec 1 sec 5 sec 8 secTime

InputEvent

Replayer

What if system is slow?

0 sec 0.5 sec 1.0 sec

Need feedback of when the system responds

1.5 sec Time

What should happen?

0 sec 0.5 sec 1.0 sec

Need feedback of when the system responds

1.5 sec Time

Workarounds?

• Current tools allow user to insert delay statements to compensate for slowness

– Creates unrealistic user behavior

– Tedious and error-prone

VNCplay approach

• Big problem is mouse clicks

• Take a screen snapshot at each mouse click during recording

• Wait for the same thing during replay

VNCplay approach

0 sec 1 sec 5 sec 8 secTime

InputEvent

Screen updates

Recorder

VNCplay approach

Screen updates

0 sec 1 sec 5 sec 8 secTime

InputEvent

Replayer

VNCplay details

• Only snapshot the screen around the cursor– Reduces trace file size– Ignores irrelevant things (e.g. system clock)

• Other tricks described in the paper

Analyzing Replays

Time

Response Time

System 1:

System 2:

Match

Evaluation

• OpenOffice on Linux

• Scaled CPU frequency from 2.0 GHz down to 300 MHz

• Compare total running time and interactive response time

Total Runtime

0

100

200

300

400

To

tal

Ru

nti

me

(sec

on

ds)

300 MHz 800 MHz 1.2 GHz 1.6 GHz 2.0 GHz

Median Response Time

050

100150200250300350400450500550600650

Res

po

nse

Tim

e (m

illi

seco

nd

s)

300 MHz 800 MHz 1.2 GHz 1.6 GHz 2.0 GHz

Other experiments

• Paper has details on more experiments

– Microsoft Windows, Linux

– Evaluate interactive performance of different disk IO schedulers

Experience using VNCplay

• Platform independence– Measured interactive performance of

PowerPoint, Word, OpenOffice

• Reliable replay– Replayed 5-minute PowerPoint session on a

slow system, taking over an hour – works OK!

Caveats

• Near-perfect reproducibility needed

– VMware: start the same VM every time

– In Linux, reset user account (dotfiles)

Caveats

• Keyboard input can be a problem– Use mouse instead of keyboard shortcuts– Click before and after you type

• Couple of tries to get a reliable session– Tooltips, pop-ups get in the way

Using VNCplay

• vncplay record server:5901 trace.vnc

• vncplay play server1:5901 trace.vnc out1.rfb• vncplay play server2:5901 trace.vnc out2.rfb

• vncplay analyze out1.rfb out2.rfb > analyze.out

• vncanalyze median analyze.out• vncanalyze cdf analyze.out > cdf.plot

Use gnuplot, Excel to plot cdf.plot

Other uses

• Automated testing– Make sure your application behaves the same

way with a new release of Linux / Windows

• GUI ``automation’’– Automate GUI-intensive tasks that have no

scriptable interface

Future directions

• Integrate some GUI toolkit knowledge for smarter matching (e.g. ignore tooltips)

• Better keyboard input handling– Machine learning– Details in paper

• If you want to help, let us know!

In conclusion …

• Need to measure interactive performance– Runtime benchmarks are not appropriate

• VNCplay– Portable, reliable session replay– Response time measurement

Questions?

http://suif.stanford.edu/vncplay/