Nghiên cứu KH-C5
Transcript of Nghiên cứu KH-C5
-
7/27/2019 Nghin cu KH-C5
1/29
CHAPTER 5
MEASUREMENT, RELIABILITY ANDVALIDITY
Nguyen Tien Dzung
Hanoi University of Science and Technology
Email: [email protected]
Website: http://dungnt.tk
-
7/27/2019 Nghin cu KH-C5
2/29
CHAPTER OVERVIEW
1. The Measurement Process2. Levels of Measurement
3. Reliability and Validity: Why They Are Very, Very Important
4. A Conceptual Definition of Reliability
5. Validity6. The Relationship Between Reliability and Validity
7. A Closing (Very Important) Thought
2012 NTD 2Research Methods - Chapter 5
-
7/27/2019 Nghin cu KH-C5
3/29
1. The Measurement Process
Two definitions Stevensassignment of numerals to objects or events
according to rules.
the assignment of values to outcomes.
2012 NTD 3Research Methods - Chapter 5
-
7/27/2019 Nghin cu KH-C5
4/29
2. Levels of Measurement
Variables are measured at one of these four levels
Qualities of one level are characteristic of the next level up
The more precise (higher) the level of measurement, the more accurateis the measurement process
Level ofMeasurement For example Quality of Level
Ratio A is 5 10 and B is 5 5 Absolute zero
Interval A is 5 taller than B An inch is an inch is an inch
Ordinal A is taller than B Greater than
Nominal A is tall and B is short Different from
2012 NTD 4Research Methods - Chapter 5
-
7/27/2019 Nghin cu KH-C5
5/29
Nominal Scale
Qualities Example What YouCan Say
What You Cant
Say
Assignment oflabels
Gender(male orfemale)
Preference(like or dislike)
Voting record(for oragainst)
Eachobservationbelongs in its
own category
An observationrepresents moreor less than
anotherobservation
2012 NTD 5Research Methods - Chapter 5
-
7/27/2019 Nghin cu KH-C5
6/29
Ordinal Scale
Qualities Example What YouCan Say
What You Cant
Say
Assignment ofvalues alongsome underlying
dimension
Rank in collegeOrder of finishing arace
Oneobservation isranked above
or belowanother.
The amount thatone variable ismore or less than
another
2012 NTD 6Research Methods - Chapter 5
-
7/27/2019 Nghin cu KH-C5
7/29
Interval Scale
Qualities Example What YouCan Say
What You Cant
Say
Equal distancesbetween points
Number of wordsspelled correctlyIntelligence test
scoresTemperature
One scorediffers fromanother on
somemeasure thathas equallyappearingintervals
The amount ofdifference is anexact
representation ofdifferences onthe variablebeing studied
2012 NTD 7Research Methods - Chapter 5
-
7/27/2019 Nghin cu KH-C5
8/29
Ratio Scale
Qualities Example What You CanSay
What You Cant
Say
Meaningful andnon-arbitraryzero
AgeWeightTime
One value istwice as muchas another orno quantity ofthat variablecan exist
Not much!
2012 NTD 8Research Methods - Chapter 5
-
7/27/2019 Nghin cu KH-C5
9/29
Continuous vs. Discrete Variables
Continuous Variables: can assume any value along some underlying continuum.
Examples: length, time, revenues, income
Discrete Variables:
Categorical variables
Those with values that can be placed into certain
categories that have definite boundaries. Examples: gender, customer satisfaction (scale from 1 to
5)
2012 NTD Research Methods - Chapter 5 9
-
7/27/2019 Nghin cu KH-C5
10/29
What Is All The Fuss?
Measurement should be as precise as possible
In behavioral and social science, most variables are
probably measured at the nominal or ordinal level.
2012 NTD 10Research Methods - Chapter 5
-
7/27/2019 Nghin cu KH-C5
11/29
3. Reliability and Validity
Importance of reliability and validity The methods and tools we use to measure the interested
object influence on the research accuracy.
Reliability: ability to produce the same results for
different measurements.Validity: ability to measure what we want to measure.
2012 NTD Research Methods - Chapter 5 11
-
7/27/2019 Nghin cu KH-C5
12/29
Reliability
ObservedScore
TrueScore
Error
2012 NTD Research Methods - Chapter 5 12
TraitError
MethodError
Error
-
7/27/2019 Nghin cu KH-C5
13/29
A Conceptual Definition Of Reliability
Method error is due to characteristics of the test or testing situation
Trait error is due to individual characteristics
Reliability of the observed score becomes higher if error is reduced!!
Method ErrorObserved Score = True Score + Error Score
Trait Error
2012 NTD 13Research Methods - Chapter 5
-
7/27/2019 Nghin cu KH-C5
14/29
INCREASING RELIABILITY
Decreasing Error
Increase sample size
Eliminate unclear questions
Standardize testing conditions
Use both easy and difficult questions
Minimize the effects of external events
Standardize instructions
Maintain consistent scoring procedures
2012 NTD 14Research Methods - Chapter 5
-
7/27/2019 Nghin cu KH-C5
15/29
How Reliability Is Measured
Reliability is measured using a Correlation coefficient
r test1test2
Reliability coefficients
Indicate how scores on one test change relative to scoreson a second test
Can range from -1.0 to +1.0 (perfect reliability)
2012 NTD 15Research Methods - Chapter 5
-
7/27/2019 Nghin cu KH-C5
16/29
Types of Reliability
Type of
Reliability
What It Is How You Do It What the Reliability
CoefficientLooks Like
Test-Retest A measure of stability Administer the same test/measureat two different times to the samegroup of participants
rtest1test1
Parallel Forms A measure ofequivalence
Administer two different forms ofthe same test to the same groupof participants
rform1form2
Inter-Rater A measure ofagreement
Have two raters rate behaviorsand then determine the amountof agreement between them
Percentage of agreements
InternalConsistency
A measure of howconsistently eachitem measures thesame underlyingconstruct
Correlate performance on eachitem with overall performanceacross participants
Cronbachs alpha
2012 NTD 16Research Methods - Chapter 5
-
7/27/2019 Nghin cu KH-C5
17/29
Validity
A valid test does what it was designed to doA valid test measures what it was designed to
measure
2012 NTD 17Research Methods - Chapter 5
-
7/27/2019 Nghin cu KH-C5
18/29
A Conceptual Definition of Validity
Validity refers to the tests results, not to the test itself
Validity ranges from low to high, it is not either/or
Validity must be interpreted within the testing context
E.g.: Calculation: 2 + 7 = ?
Which ability does the task measure?
Addition abilityor Multiplication ability or Mathematical ability?
2012 NTD 18Research Methods - Chapter 5
-
7/27/2019 Nghin cu KH-C5
19/29
Validity
Validity tool measures what-it-should Example:
Question: 2 + 3 = 5
I check 10 times with ten addition calculations, you answer 100%correctly your answer is very reliable.
I use the test to measure your intelligence. Is the test valid?
NO. BECAUSE THIS TEST IS VALID ONLY FOR MEASURINGADDITION ABILITY, NOT MORE THAN THAT; INTELLIGENCE IS
VERY COMPLICATED ABILITY.
2012 NTD 19Research Methods - Chapter 5
-
7/27/2019 Nghin cu KH-C5
20/29
Example of Validity and Reliability
A wife wants to measure the loyalty of her husband She uses a criterion: never come late
He actually never comes home late.
Is the test valid?
2012 NTD 20Research Methods - Chapter 5
-
7/27/2019 Nghin cu KH-C5
21/29
Types of Validity
Type ofValidity
What Is It? How Do You Establish It?
Content A measure of how well theitems represent the entireuniverse of items
Ask an expert if the items assesswhat you want them to
Criterion
Concurrent A measure of how well atest estimates a criterion
Select a criterion and correlatescores on the test with scores onthe criterion in the present
Predictive A measure of how well atest predicts a criterion
Select a criterion and correlatescores on the test with scores on
the criterion in the futureConstruct A measure of how well a
test assesses someunderlying construct
Assess the underlying construct onwhich the test is based andcorrelate these scores with thetest scores
2012 NTD 21Research Methods - Chapter 5
-
7/27/2019 Nghin cu KH-C5
22/29
Content Validity
A measure of how well the items represent theentire universe of items
Ask experts to check whether the items assesswhat you want them to access
E.g.:
You want to measure education quality
You must ask experts. They show you some relevant
criteria to measure (teachers, libraries, facilities,curricula, research activities and applications, reputation)
2012 NTD 22Research Methods - Chapter 5
-
7/27/2019 Nghin cu KH-C5
23/29
Exercise for homework
Project 1:You want to measure: personal intelligence
How to define the concept? (What are the criteria ofintelligence?)
Project 2:
You want to measure: personal happiness
How to measure?
2012 NTD 23Research Methods - Chapter 5
-
7/27/2019 Nghin cu KH-C5
24/29
Criterion Validity
Concerned with how well a test either estimates performance (concurrent validity) or
how well a test predicts future performance (predictive vadility)
Example:
A research to measure how GPA (Grade Point in Average) of
graduate students relate to their research ability
The research ability is measured by the number of articles publishedon journals
If the correlation between GPA and the number of published articlesis high, it means GPA is a good predictor of the number of publishedarticles (high predictive validity)
Does a high number of published articles mean a good researchability? is the question about concurrent validity of the research
2012 NTD 24Research Methods - Chapter 5
-
7/27/2019 Nghin cu KH-C5
25/29
How To Establish Construct Validity of
A New Test
Correlate new test with an established test
Show that people with and without certain traitsscore differently
Determine whether tasks required on test areconsistent with theory guiding test development
2012 NTD 25Research Methods - Chapter 5
-
7/27/2019 Nghin cu KH-C5
26/29
Multitrait-multimethod Matrix
2 research projects about children: Project 1: To measure impulsivity (Trait 1)
Project 2: To measure activity level or movement (Trait 2)
To test a construct validity, use multitrait-
multimethod matrix Project 1:
Method 1: paper-and-pencil instrument on childs wrists
Method 2: activity meter (e.g.: observation)
Project 2: Method 1: paper-and-pencil instrument on childs wrists Method 2: activity meter (e.g.: observation)
2012 NTD 26Research Methods - Chapter 5
-
7/27/2019 Nghin cu KH-C5
27/29
MULTITRAIT-MULITMETHOD MATRIX
Convergent validitydifferent methods yield similar results
Discriminant validitydifferent methods yield different results
Method 1
Paper andPencil
Method 2
Activity LevelMonitor
Method 1
Paper andPencil
Method 2
Activity LevelMonitor
Trait 1
Method 1
Paper and Pencil Moderate Low
Impulsivity Method 2Activity Level
Monitor
Moderate
Trait 2
Method 1
Paper and Pencil
Activity
Level
Method 2
Activity LevelMonitor
Low
Trait 1
Impulsivity
Trait 2
Activity Level
2012 NTD 27Research Methods - Chapter 5
-
7/27/2019 Nghin cu KH-C5
28/29
The Relationship Between Reliability and Validity
(The Books Point Of View)
A valid test must be reliableBut
A reliable test may not be valid
A high reliability is a necessity to have a high validity, but notensure
2012 NTD 28Research Methods - Chapter 5
VALIDITY AND RELIABILITY (Th t h i t f
-
7/27/2019 Nghin cu KH-C5
29/29
VALIDITY AND RELIABILITY (The teachers point of
view)
a) High validity and high reliability b) High validity but low reliability
c) Low validity but high reliability d) Low validity and low reliability 2012 NTD 29Research Methods Chapter 5