PERFECTING THE TEST THROUGH QUESTION … THE TEST THROUGH QUESTION ANALYSIS ... More than 41,000...
13
1 2010 Users Conference Miami PERFECTING THE TEST THROUGH QUESTION ANALYSIS DELL CONFIDENTIAL ABOUT THE PRESENTER NEELOV KAR Program Owner for Project Management courses Led the Evaluation Team and introduced QM in Perot Systems Account Manager in Perot Systems since 1998 Worked in other organizations in senior positions and has a total of 22 years of IT experience PMP, ISO 9000 Lead Assessor, Six Sigma and Certified Quality Analyst Certified Instructor for Orientation, Leadership Workshop and Project Management courses. Graduation from Friendship University, Moscow and MBA from BIT Mesra DELL CONFIDENTIAL
Transcript of PERFECTING THE TEST THROUGH QUESTION … THE TEST THROUGH QUESTION ANALYSIS ... More than 41,000...
1
2010 Users Conference Miami
PERFECTING THE TEST THROUGH QUESTION ANALYSIS
DELL CONFIDENTIAL
ABOUT THE PRESENTER
NEELOV KAR
Program Owner for Project Management courses
Led the Evaluation Team and introduced QM in Perot Systems
Account Manager in Perot Systems since 1998
Worked in other organizations in senior positions and has a total of 22 years of IT experience
PMP, ISO 9000 Lead Assessor, Six Sigma and Certified Quality Analyst
Certified Instructor for Orientation, Leadership Workshop and Project Management courses.
Graduation from Friendship University, Moscow and MBA from BIT Mesra
DELL CONFIDENTIAL
2
2010 Users Conference Miami
WELCOME TO DELL SERVICES
3
“Dell Services is a one stop shop for all
IT Services delivering its services
worldwide in 183 countries with an
employee strength of 41000”.
DELL CONFIDENTIAL
Building on years of successful collaboration, together we create Dell Services
Serving the public sector, large enterprises and small- and medium-size businesses all over the world
More than 41,000 experienced and skilled technology and business-services professionals
Approximately $8B revenues from enhanced services and support
Best-in-class suite of intelligent, end-to-end services and business solutions to reduce IT complexity and lower costs
4
Global Presence
DELL CONFIDENTIAL
3
5
The Integrated Services Portfolio
From product support to business solutions, we can help you
achieve your business outcomes
SupportServices
Managed Services
Cloud /as-a-Service
ApplicationsServices
Business ProcessServices
ITConsulting
BusinessConsulting
Extended Warranty
End UserClient
SoftwareCustom
DevelopmentRevenue Cycle
OutsourcingEnterprise
ArchitectureStrategy
Consulting
Enhanced Support
Data CenterEnterprise
ManagementSoftware
ModernizationPolicy
AdministrationService
Management
Organizational Change
Management
NetworkBusiness Continuity Software
Business Intelligence
Physician ServicesData Center
InfrastructureProcess Re-Engineering
Information Assurance
DesktopEnterprise and
Industry Applications
Engineering Services
Outsourcing
EnterpriseApplications
Rationalization
Supply Chain Re-Engineering
HostingServers and
StorageTesting
Customer Management
End User Computing
Clinical Transformation
Management Government OfficeVirtualization/
Cloud Integration
Data Management
Business Continuity /
Disaster RecoveryDELL CONFIDENTIAL
Dell Services: Foundation of Technology
With Industry Capabilities
HealthcareGovernmentCommercial
Hospitals
Physicians
Health Insurance
Life Sciences
Pharmaceutical
Defense
Homeland Security
Intelligence
Civilian
State and Local
Education
Retail Banking
Payments and Clearing
Insurance
Capital Markets
Manufacturing
Freight and Logistics
Telecommunications
Travel and Transport
Energy
Retail
Technology
6DELL CONFIDENTIAL
4
7
Helping You Succeed
Setting Dell apart through our breadth of portfolio, ecosystem of
partners and systematic approach to innovation
Trusted by you:
Over 2 million desktops managed
12.8 million desktops and notebooks supported
1.4 million servers and storage systems supported
Over 5,000 SaaScustomers
Ranked #1 for Professional Services in North America Healthcare*
Ranked #1 in Customer Service**
Using customizable technology to deliver business-focused solutions
Leveraging internet technology, we will be at the forefront of cloud and as-a-service delivery
Using standardization to drive down costs and drive up efficiency, from the desktop to the data center
Evolving with the industry to bring you the newest technology and more flexible consumption models
*By revenue from Gartner’s Industry IT Services Worldwide Market Share Database; August 14, 2009 **TBR's Q209 Service and Support Customer
Satisfaction Study
DELL CONFIDENTIAL
Responsible for Design, Development & Delivery of Project Management and Leadership courses for Dell Services across the globe
ISO 9000 certified since 2007
Awarded “2009 Best Place to Learn” by ASTD Dallas Chapter
We use
– QM for Level 2 Assessment
– SurveyMonkey for Level 1 feedback
– Adobe Captivate for CBL development
– PeopleSoft for Training data management
– Saba as LMS
DELL CONFIDENTIAL 8
Learning & Development
5
ADDIE MODEL
DELL CONFIDENTIAL 9
Learn to use the different Perception Report to comprehensively analyze the assessment
Understand how item analysis Report helps in identifying the badly constructed questions
Learn how Dell Services have done the analysis to refine the test
DELL CONFIDENTIAL 10
Learning Objectives
6
P3MM Fundamentals Course
3 Series with 17 VILT Modules
Exam after each series
Launched in December 2008
Cohort 1, 2 & 3 as Pilot, participants were experienced project managers
Used Cohort 1, 2 & 3 Exam results and participant feedback to analyze the test questions
DELL CONFIDENTIAL 11
The Case
Done through QM
– Question Statistics Report
– Item Analysis Report
– Indices to look for:
– Discrimination Index
– Item Difficulty Level
Combination of SurveyMonkey and QM
– Correlation between Test Score and Demography (Role, Account, BU etc)
QM and Course Delivery
– Correlation between Test Score and Course Content
– Correlation between Test Score and Instructor
TEST ANALYSIS
DELL CONFIDENTIAL
7
Completed Schedule 29
80% or over 12 41%
Average time 00:20:05
Completed Cost 21
80% or over 4 19%
Average time 00:19:43
Completed Scope 21
80% or over 17 80%
Average time 00:09:54
Completed All 20
80% or over 10 50%
Average time 00:35:57
TEST RESULTS
Raised a red flag that there is some problem with Cost Test
DELL CONFIDENTIAL
Course Total
Scope Management 12 9 75% 3 25%
Cost Management 16 8 50% 6 38% 2 13%
Schedule Management 34 27 79% 1 3% 5 15% 1 3%
Majority Correct Polarized Scattered Majority Wrong
ANALYZED DISTRIBUTION OF
ANSWERS
CategoriesMajority Correct
Polarized
Scattered
Majority Wrong
DELL CONFIDENTIAL
8
DELL CONFIDENTIAL 15
Test Analysis Report
DELL CONFIDENTIAL 16
Question Statistics Report
Scattered
9
DELL CONFIDENTIAL 17
Question Statistics Report
Polarized
DELL CONFIDENTIAL 18
Item Analysis Report
10
DELL CONFIDENTIAL 19
Item Analysis Report
DELL CONFIDENTIAL 20
Item Analysis Report
11
DELL CONFIDENTIAL 21
Item Analysis Report
Analyzing the Question and Feedback
DELL CONFIDENTIAL
12
DELL CONFIDENTIAL 23
COACHING REPORT
DELL CONFIDENTIAL 24
Assessment Overview Report
13
DELL CONFIDENTIAL 25
P3MM FUNDAMENTALS COURSE HAS
BEEN STABILIZED AFTER FEBRUARY
2009 AND RUNNING SMOOTHLY SINCE
THEN………
2010 Users Conference Miami
THANK YOU
26
