Elaine Lopes - Cisco

Julie Timmcke PTC

Oct 2009

Performance Testing Tools Survey

Design & Delivery Committee

The charter for the Design and Delivery Committee is to build and share with our valued members a knowledge base of performance test design, content development process, and delivery of the performance tests, thus making the performance testing adoption easier. Getting involved

Attend the summits

Attend D&D conference calls in between summits test_design@lists.performancetest.org

Provide critiques on our Request For Comment (RFC) documents.

Join the Performance Testing Council

Tools Survey Background Information

Survey goal: Discover what tools are currently being used for performance-based testing and identify tools or functionalities that are needed but not yet available.

Preso goals:

Present survey data as is

Not meant to draw conclusions or make predictions

Numbers:

Half (9 out of 18) of the respondents responded on the 1st day

The survey was sent to 715 people

Response rate is 2.5%

40 people started it, 18 completed it started/completed ratio is 45%

the length of the survey

technical issue - loop on question 85

Confidence level = 95%

Margin of error = 22.82%

*

Answer OptionsResponse PercentResponse CountYes28.2%11No71.8%28answered question39skipped question0

Q1

Mandatory

*

Am. Board of Emergency Medicine

American Board of Anesthesiology

American Society for Quality

Applied Measurement Professionals Inc.

Applied Simulation Technologies

Cisco

FBI

Federation of State Boards of Physical Therapy

FL Nursery, Growers & Landscape Assn

Kaplan IT

LyondellBasell

Microsoft

NAC, NETPDTC, USN

NCIDQ

NKBA

Novell

Red Hat

Sun Microsystems

The Caviart Group

Toolwire

If you are not a member and would like to receive a high level summary of the survey results, please provide your contact information (*your name and email address are required). Performance Testing Council members will receive detailed survey results. Answer OptionsResponse PercentResponse CountCompany:84.6%22answered question26skipped question13

Q2

*

CEO, Chief Scientist

Certification Process Manager

Consultant

Director of Instructional Systems Design

Executive

Executive Director

Organizational Consultant

Organizational Psychologist

Test Developer

Test Developer and Program Manager

we don't have a director level role, but it's closest to what I do

Answer OptionsResponse PercentResponse CountItem Writer/Developer5.3%2Program Manager13.2%5Project Manager2.6%1IT Manager5.3%2Subject Matter Expert2.6%1Psychometrician21.1%8Marketing Manager5.3%2Quality Assurance/Audit Manager2.6%1Software Engineer/Programmer2.6%1Director of Testing7.9%3Other (please specify)31.6%12answered question38skipped question1

Q3

*

Answer OptionsResponse PercentResponse CountLess than 1 year5.3%21 year to 3 years21.1%83 years to 7 years28.9%117 years to 10 years15.8%610 years or more28.9%11answered question38skipped question1

Q4

*

Answer OptionsResponse PercentResponse Count1 to 1003.2%1101 to 1,2009.7%31,201 to 5,00012.9%45,001 to 10,00022.6%710,001 to 25,0006.5%225,001 to 100,00022.6%7100,000 or more22.6%7answered question31skipped question8

Q5

935,124 annual test delivery instances considering the lower number

*

Answer OptionsResponse PercentResponse CountYes80.6%25No9.7%3Not sure9.7%3answered question31skipped question8

Q6

*

MySQL DBA 5.1NumerousStorage virtualizationThese certifications belong to our clients. I don't have permission to share them. By the way, the survey assumes that the respondent is a testing program and not a vendor. Moreover it assumes that the testing program is a certification one. Obviously, there are programs in other areas such as education, admissions, licensure, I/O.NCCCO, ASNT, CrossFit, NAECReal Estate Licensing - 7 different states; Strength and Conditioning; OrthoticsSun Certified Enterprise Architect, Sun Certified Java DeveloperNovell Certified Engineer - Enterprise Services, Novell Certified Linux Professional, Novell Certified Linux Engineer70-620HP Certifications for LoadrunnerFNGLA Certified Landscape Technician, FNGLA Certified Landscape Maintenance Technician"Oral certification examinationInitial Certification in AnesthesiologyHealth, Professional, ITAdvancement exam for musicians includes some multimedia performance-based items (audio and video files).All Microsoft Office 2003 and 2007 certifications, Microsoft Certified Systems EngineerMicrosoft Certified Systems Administrator, Microsoft Certified IT Professional: Windows Server 2008 Server Administrator, Microsoft Certified IT Professional: Windows Server 2008 Enterprise Administrator, Microsoft Certified Technical Specialist: Windows Server 2008 Active Directory, Configuring, Microsoft Certified Technical Specialist: Microsoft SQL Server 2005, Microsoft Certified Technical Specialist: Windows Vista Configuration"OperatorsIP Office Performance LabAll of our exam programs use performance based items.CCIE lab exams. Associate and Professional exams also incorporate PBT items via simulations.Certified Kitchen Designer, Certified Bath DesignerNCIDQ Certificate

If yes, please list those certifications that employ any performance items.

Q7

*

Answer OptionsResponse PercentResponse Count1 to 10014.3%4101 to 1,20025.0%71,201 to 5,00014.3%45,001 to 10,00010.7%310,001 to 25,00017.9%525,001 to 100,00010.7%3100,000 or more7.1%2answered question28skipped question11

Q8

345,526 annual performance test delivery instances considering the lower number

*

Many of our exams use multiple choice options merely as a method of gathering a response. The response had to be derived from one or more performances. I'm confused as to why complex behaviors measured by text/graphics/video/audio-based multiple choice questions are not included in the list. There are many ways to measure performance than those listed above.

Proctorless, automated evaluation, determinining end result (not methods or tasks to get there) to score exam objectives.

Board oral exam using simulated cases

Design scenarios graded by trained interior designers

Considering just those testing programs employing performance based items, what design approaches are you using? Check all that apply.Answer OptionsResponse PercentResponse CountLive Environment46.4%13Monitored Hands-On Exam46.4%13Simulating Software using Software25.0%7Simulating Hardware Using Software17.9%5Simulating Hardware using Hardware AND Software 10.7%3Simulating Hardware or Environment using Video10.7%3Simulating Hardware using 3-D Models 3.6%1Simulating Process using Software14.3%4Emulation 28.6%8Adaptive Text Case Scenario or Adaptive Case Study28.6%8Audio Exam21.4%6Board Assessment/Oral Exam14.3%4Other (please describe)14.3%4answered question28skipped question11

Q9

*

Q9- contd

*

Answer OptionsResponse PercentResponse CountYes58.1%18No29.0%9Do not know12.9%4answered question31skipped question8

Q10

Mandatory

Logic: Q11, Q19, Q27, Q35, Q43

*

Chart5YesNoDo not knowConsidering your exams employing performance-based items, does your organization use any tools in the Item Development category? Item Development tools include Content Authoring, Standard Setting, Simulation Software, Emulation Software and Item Bank Management.0.5810.290.129Question 1Performance Testing ToolsPerformance Testing ToolsPerformance Testing ToolsPerformance Testing ToolsAre you a member of the Performance Testing Council?Are you a member of the Performance Testing Council?Are you a member of the Performance Testing Council?Are you a member of the Performance Testing Council?Answer OptionsAnswer OptionsResponse PercentResponse CountYesYes28.2%11NoNo71.8%28answered questionanswered question3939skipped questionskipped question00Question 100Are you a member of the Performance Testing Council?Question 2Performance Testing ToolsPerformance Testing ToolsPerformance Testing ToolsPerformance Testing ToolsIf you are not a member and would like to receive a high level summary of the survey results, please provide your contact information (*your name and email address are required). Performance Testing Council members will receive detailed survey results.If you are not a member and would like to receive a high level summary of the survey results, please provide your contact information (*your name and email address are required). Performance Testing Council members will receive detailed survey results.If you are not a member and would like to receive a high level summary of the survey results, please provide your contact information (*your name and email address are required). Performance Testing Council members will receive detailed survey results.If you are not a member and would like to receive a high level summary of the survey results, please provide your contact information (*your name and email address are required). Performance Testing Council members will receive detailed survey results.Answer OptionsAnswer OptionsResponse PercentResponse Count*Name:*Name:100.0%26Company:Company:84.6%22*Email Address:*Email Address:100.0%26Phone:Phone:61.5%16answered questionanswered question2626skipped questionskipped question1313NumberResponse Date*Name:Company:*Email Address:Phone:1Sep 22, 2009 8:50 PMBuck ChaffeeThe Caviart Groupcchaffee@thecaviartgroup.com703-424-85132Sep 22, 2009 8:53 PMCynthia SearcyFederation of State Boards of Physical Therapycsearcy@fsbpt.org(703) 739-9420 x2063Sep 22, 2009 8:55 PMDavid StokesSun Microsystemsdavid.stokes@sun.com817-210-64864Sep 22, 2009 8:59 PMDiane HuntsingerDianeh@netapp.com5Sep 22, 2009 9:21 PMaa6Sep 22, 2009 10:02 PMljensen@redhat.comRed Hatljensen@redhat.com703-506-94947Sep 22, 2009 10:28 PMWilliam D. HoganApplied Measurement Professionals Inc.bhogan2@earthlink.net830-833-16948Sep 22, 2009 11:14 PMSean GaleNovellsean@seanalicia.com9Sep 23, 2009 12:43 AMGraeme JohnstonToolwiregjohnston@toolwire.com508308087510Sep 23, 2009 1:41 AMMichael BestKaplan ITmike.best@kaplan.com11Sep 23, 2009 1:04 PMMerry MottFL Nursery, Growers & Landscape Assnmmott@fngla.org407-295-799412Sep 23, 2009 1:12 PMBob KorteAm. Board of Emergency Medicinerkorte@abem.org517.332.4800 ext. 33613Sep 23, 2009 2:02 PMJoseph McClintockAmerican Board of Anesthesiologyjoe.mcclintock@theaba.org919-573-431514Sep 23, 2009 6:06 PML. Kirk SchultzNAC, NETPDTC, USNlee.schultz@navy.mil15Sep 23, 2009 6:07 PMtony zaratony.zara@pearson.com16Sep 23, 2009 9:48 PMLiberty MunsonMicrosoftliberty.munson@microsoft.com17Sep 24, 2009 10:48 AMRick RepnakLyondellBasellrick.repnak@lyondellbasell.com713-401-219218Sep 24, 2009 12:00 PMBR Dawadibdawadi@smttest.com19Sep 25, 2009 1:54 AMCamille B. PriceApplied Simulation Technologiescprice@appliedsimtech.com801-506-134020Sep 26, 2009 12:56 PMCarmen O'NeillAmerican Society for Qualityconeill@asq.org1-800-248-194621Sep 27, 2009 2:55 PMJennifer HurdFBIjennifer.hurd@ic.fbi.gov202-324-392122Sep 29, 2009 2:00 PMCarmen O NeillAmerican Society for Qualityconeill@asq.org800248194623Sep 30, 2009 12:12 AMJennifer HurdFBIj_m_hurd@yahoo.com24Sep 30, 2009 9:52 PMSarah DeMarkCiscosdemark@cisco.com25Oct 5, 2009 1:47 PMVeronica WhiteheadNKBAvwhitehead@nkba.org908-813-484626Oct 6, 2009 3:42 PMJeff KenneyNCIDQjeff@ncidq.org202-721-0220Question 20000If you are not a member and would like to receive a high level summary of the survey results, please provide your contact information (*your name and email address are required). Performance Testing Council members will receive detailed survey results.Question 3Performance Testing ToolsPerformance Testing ToolsPerformance Testing ToolsPerformance Testing ToolsWhat is your primary job role?What is your primary job role?What is your primary job role?What is your primary job role?Answer OptionsAnswer OptionsResponse PercentResponse CountContent ManagerContent Manager0.0%0Item Writer/DeveloperItem Writer/Developer5.3%2Program ManagerProgram Manager13.2%5Project ManagerProject Manager2.6%1IT ManagerIT Manager5.3%2Job AnalystJob Analyst0.0%0Subject Matter ExpertSubject Matter Expert2.6%1PsychometricianPsychometrician21.1%8Marketing ManagerMarketing Manager5.3%2Customer Service RepresentativeCustomer Service Representative0.0%0Quality Assurance/Audit ManagerQuality Assurance/Audit Manager2.6%1Software Engineer/ProgrammerSoftware Engineer/Programmer2.6%1ProctorProctor0.0%0AttorneyAttorney0.0%0Technical EditorTechnical Editor0.0%0Instructional DesignerInstructional Designer0.0%0Director of TestingDirector of Testing7.9%3Language EditorLanguage Editor0.0%0Other (please specify)Other (please specify)31.6%12answered questionanswered question3838skipped questionskipped question11NumberResponse DateOther (please specify)1Sep 22, 2009 8:50 PMConsultant2Sep 22, 2009 9:02 PMCEO, Chief Scientist3Sep 22, 2009 10:03 PMCertification Process Manager4Sep 22, 2009 10:54 PMwe don't have a director level role, but it's closest to what I do.5Sep 23, 2009 3:19 PMExecutive6Sep 23, 2009 9:27 PMTest Developer and Program Manager7Sep 25, 2009 1:56 AMDirector of Instructional Systems Design8Sep 26, 2009 12:57 PMTest Developer9Sep 27, 2009 2:57 PMOrganizational Consultant10Sep 29, 2009 2:00 PMTest Developer11Sep 30, 2009 12:13 AMOrganizational Psychologist12Oct 6, 2009 3:43 PMExecutive DirectorQuestion 30000000000000000000What is your primary job role?Question 4Performance Testing ToolsPerformance Testing ToolsPerformance Testing ToolsPerformance Testing ToolsHow long have you been in this role?How long have you been in this role?How long have you been in this role?How long have you been in this role?Answer OptionsAnswer OptionsResponse PercentResponse CountLess than 1 yearLess than 1 year5.3%21 year to 3 years1 year to 3 years21.1%83 years to 7 years3 years to 7 years28.9%117 years to 10 years7 years to 10 years15.8%610 years or more10 years or more28.9%11answered questionanswered question3838skipped questionskipped question11Question 400000How long have you been in this role?Question 5Performance Testing ToolsPerformance Testing ToolsPerformance Testing ToolsPerformance Testing ToolsConsidering all exams in your organization's testing programs, how many test delivery instances are administered annually (not individual candidates but actual number of tests taken regardless of outcome)?Considering all exams in your organization's testing programs, how many test delivery instances are administered annually (not individual candidates but actual number of tests taken regardless of outcome)?Considering all exams in your organization's testing programs, how many test delivery instances are administered annually (not individual candidates but actual number of tests taken regardless of outcome)?Considering all exams in your organization's testing programs, how many test delivery instances are administered annually (not individual candidates but actual number of tests taken regardless of outcome)?Answer OptionsAnswer OptionsResponse PercentResponse Count1 to 1001 to 1003.2%1101 to 1,200101 to 1,2009.7%31,201 to 5,0001,201 to 5,00012.9%45,001 to 10,0005,001 to 10,00022.6%710,001 to 25,00010,001 to 25,000...