Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
2011 European Users Conference Brussels
Automated testing of assessment delivery performance
using Jmeter
Michel Duijvestijn
Rotterdam University of Applied Science
2011 European Users Conference Brussels Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
How YOU(!) can use a tool (JMeter) to test the delivery of QMP-assessments
Why you may want to do that
How it works and why it‟s important to know a bit about that
What parameters there are to set
How to interprete the test results
How we set up a complete testing environment
Our results until now
What to expect?
2011 European Users Conference Brussels Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Introduction
How to get started
Running the script as a black box
How to interprete the test results
Into the black box: how things work
Parameter settings
Setting up an integrated test environment
Some test results
Discussion
Agenda
2011 European Users Conference Brussels Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Software-tool to build and execute JMeter testscripts
JMeter testscript can be used to test webapplications (like QMP assessment delivery)
Open source
Java based
Multi-platform (windows, linux, Mac?)
Very active community
What is Jmeter?
2011 European Users Conference Brussels Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
With a proper testscript you can simulate an assessment delivery in QMP: „Student‟ goes to website
Is logging on
Chooses the right assessment
Gives a „valid‟ answer to all of the questions
takes a look at the feedback
Returns to the homepage
Is logging off again
In connection to QMP:
2011 European Users Conference Brussels Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Performance: How many concurrent users
While using random questions: all questions performing well?
May be helpfull for debugging incidently occurring questions
Regression test after server maintenance?
Controlling 24x7 server availability
Possible applications
2011 European Users Conference Brussels Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Introduction
How to get started Running the script as a black box
How to interprete the test results
Into the black box: how things work
Parameter settings
Setting up an integrated test environment
Some test results
Discussion
Agenda
2011 European Users Conference Brussels Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
An all-inclusive ZIP-file containing JMeter installation
“working Demo” script for QMP version 5
A conversion of this script to work with QMP v4 (NB: with limited number of question types!)
“getting started” guide for this script
“reference guide” with complete documentation
Extended script for use in combination with…
…the “work in progress” Access database to control the proces (NB: only partly documented)
Support as much as possible, but no guarantee!
What do I have to offer?
Slide 8
2011 European Users Conference Brussels Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Schedule assessment in QMP
Unzip the file (no need to install) Optional: plugin
Start JMeter.bat and load the demoscript
Provide script with your own settings
Run it!
To get started
DEMO…
2011 European Users Conference Brussels Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Import group of testusers into QMP (all having the same password)
Make a copy of the assessment to test
Show feedback at end of assessment
Make a note of Assessment ID (see assessment properties)
Schedule Assessment (JM-version) for testusers
Preparations within QMP
2011 European Users Conference Brussels Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
JMeter software in C:\jakarta-jmeter-2.4?
Plugin-jar in C:\jakarta-jmeter-2.4\lib\ext?
JMeterDemo in \bin\scripts?
Link on desktop to \bin\scripts\JMeter.bat (including –t-link to script?!)
File with usernames at proper spot
Other preparations
2011 European Users Conference Brussels Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
„testplan‟-level sCustomerid (YOURID) (= repository) Path (/your/serverpath/perception.php) Password (YourParticipantPassword)
„HTTP-request defaults level Server url (your.perceptionserver.nl) Http protocol + poort (http/https; 80/443)
User parameters level Assessment-id (at sSession) Assessment-name (optional) Check path to file with user names
Adjust script settings
2011 European Users Conference Brussels Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Introduction
How to get started
Running the script as a black box
How to interprete the test results
Into the black box: how things work
Parameter settings
Setting up an integrated test environment
Some test results
Discussion
Agenda
2011 European Users Conference Brussels Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
<ctrl+e> clear results
<ctrl+r> run
Start it
Slide 14
2011 European Users Conference Brussels Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Following progress with View Results Tree
Slide 15
2011 European Users Conference Brussels Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Following progress with Aggregate report
Slide 16
2011 European Users Conference Brussels Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
<ctrl+.>
If things go wrong: STOP!
Slide 17
2011 European Users Conference Brussels Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Introduction
How to get started
Running the script as a black box
How to interprete the test results
Into the black box: how things work
Parameter settings
Setting up an integrated test environment
Some test results
Discussion
Agenda
2011 European Users Conference Brussels Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Green response? Not necessarily everything OK!
So: check if every step gives the right response!
No errors???
Slide 19
2011 European Users Conference Brussels Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Shows responstime in ms
Can rise to tens of seconds!
Response time dependent of assessment timing (more about that later)
The complexity of the assessment
Of course number of concurrent users
Important: Error%!
Interpretation „aggregate report‟
2011 European Users Conference Brussels Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Introduction
How to get started
Running the script as a black box
How to interprete the test results
Into the black box: how things work
Parameter settings
Setting up an integrated test environment
Some test results
Discussion
Agenda
2011 European Users Conference Brussels Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Listeners (show results) View result tree (result of individual requests) Aggregate report (response times, errors)
Controllers (logical structure) If/then While
Samplers http-request Debug sampler
Timers
Error-handling: assertions
Parts to discuss
2011 European Users Conference Brussels Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
To start an assessment you need a set of parameters The good news: this information is available in the
previous page!
The magic: how to get this information out of the page?
How to start an assessment?
customerid=HSRPROD&
action=StartAssessment&
login=5637881790119879316&
authentication=Password&
progress=0&
syslang=nl-NL&
assessmentID=7939180316566014&
lang=-&
expires=2011-03-05T18:09:43.818186+01:00&
resume=false&
access=CEC090C693D0D5C330468B0BC967B27C&
monitor=false&
useQS=false&
isOpen=false&
assessmentName=JM_Stresstest - Stoas&
scheduleID=864719902&
scheduleGroupID=161686849&
forDownload=false
customerid=HSRPROD">
<input name="action" type="hidden" value="StartAssessment">
<input name="login" type="hidden" value="5637881790119879316">
<input name="authentication" type="hidden" value="Password">
<input name="progress" type="hidden" value="0">
<input name="syslang" type="hidden" value="nl-NL">
<input name="assessmentID" type="hidden" value="7939180316566014">
<input name="lang" type="hidden" value="-">
<input name="expires" type="hidden" value="2011-03-05T18:09:43.818186+01:00">
<input name="resume" type="hidden" value="false">
<input name="access" type="hidden" value="CEC090C693D0D5C330468B0BC967B27C">
<input name="monitor" type="hidden" value="false">
<input name="useQS" type="hidden" value="false">
<input name="isOpen" type="hidden" value="false">
<input name="assessmentName" type="hidden" value="JM_Stresstest - Stoas">
<input name="scheduleID" type="hidden" value="864719902">
<input name="scheduleGroupID" type="hidden" value="161686849">
<input name="forDownload" type="hidden" value="false“>
2011 European Users Conference Brussels Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Answer: regular expression extractor
2011 European Users Conference Brussels Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Retrieve information with “post-processor” at loginpage…
2011 European Users Conference Brussels Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
… and apply it with the request for the next page (= assessment)
2011 European Users Conference Brussels Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
The same way: determining which answers to give
2011 European Users Conference Brussels Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
QMP generating an error message = for JMeter correct received data, so nothing wrong?!
Solution: define when JMeter shoud “fail” : assertions
If things go wrong within QMP: response assertion
2011 European Users Conference Brussels Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Yes and no…
You can use it as a black box but:
You may need to dig in the code in case anything is not working properly
Extensive reference guide available (at community website)
Sounds complicated? Do I need to know?
2011 European Users Conference Brussels Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Introduction
How to get started
Running the script as a black box
How to interprete the test results
Into the black box: how things work
Parameter settings Setting up an integrated test environment
Some test results
Discussion
Agenda
2011 European Users Conference Brussels Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Without measurements: everything ASAP = how to crash your server with DDoS
Not a good representation of the actual situation
Solution: use timers Constant delay offset
Gaussian: constant delay and deviation
Start of the test: „ramp-up‟ time (= time in seconds within all threads should start up)
Assessment timing
2011 European Users Conference Brussels Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
2011 European Users Conference Brussels Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Introduction
How to get started
Running the script as a black box
How to interprete the test results
Into the black box: how things work
Parameter settings
Setting up an integrated test environment
Some test results
Discussion
Agenda
2011 European Users Conference Brussels Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Manually changing parameters
No storage of results
In case of parallel use: no combination of results
limitations of basic script
Slide 34
2011 European Users Conference Brussels Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
New script version Same basic principle, but
All input and output via text files
Some adaptations for „parallel use‟ (max. 12 x 20)
Access database Parameter preset tables
VBA code to export necessary parameter files
VBA code to monitor progress
VBA code to import results
Reports to display results
Available, but only partly documented
Integrated test environment (how we dit it)
Slide 35
2011 European Users Conference Brussels Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
2011 European Users Conference Brussels Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Select testparameters
Slide 37
2011 European Users Conference Brussels Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Monitoring progress
Slide 38
2011 European Users Conference Brussels Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Display results
Slide 39
2011 European Users Conference Brussels Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Introduction
How to get started
Running the script as a black box
How to interprete the test results
Into the black box: how things work
Parameter settings
Setting up an integrated test environment
Some test results Discussion
Agenda
2011 European Users Conference Brussels Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Some results (1)
2011 European Users Conference Brussels Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Some results (2)
2011 European Users Conference Brussels Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Introduction
How to get started
Running the script as a black box
How to interprete the test results
Into the black box: how things work
Parameter settings
Setting up an integrated test environment
Some test results
Discussion
Agenda
2011 European Users Conference Brussels Copyright © 1995-2011 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved.
Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Questions…
Top Related