Review: Performance-Based Assessments

• Performanc-based assessment• Real-life setting

• H.O.T.S.

• Techniques:• Observation• Individual or Group Projects• Portfolios• Performances• Student Logs or Journals

• Developing performance-based assessments• Determining the purpose of assessment

• Deciding what constitutes student learning

• Selecting the appropriate assessment task

• Setting performance criteria

Review: Grading

• Grading process:

• Making grading fair, reliable, and valid• Determine defensible objectives• Ability group students• Construct tests which reflect objectivity• No test is perfectly reliable• Grades should reflect status, not improvement• Do not use grades to reward good effort• Consider grades as measurements, not evaluations

Objectivesof instruction

Test selectionand administration

Results comparedto standards

Finalgrades

Cognitive Assessments

PhysicalFitnessKnowledge

HPHE 3150Dr. Ayers

PhysicalFitnessKnowledge

Test Planning

• TypesMastery (driver’s license)

Meet minimum requirements

Achievement (mid-term)

Discriminate among levels of accomplishment

Table of Specifications(content-related validity)

• Content Objectiveshistory, values, equipment, etiquette, safety, rules, strategy, techniques of play

• Educational Objectives (Blooms’ taxonomy, 1956)

knowledge, comprehension, application, analysis, synthesis, evaluation

Table of Specifications for a 33Item Exercise Physiology Concepts Test

(Ask-PE, Ayers, 2003)

T of SPECS-E.doc

Test Characteristics

• When to test• Often enough for reliability but not too often to be useless

• How many questions (p. 145-6 guidelines)• More items yield greater reliability

• Format to use (p. 147 guidelines)• Oral (NO), group (NO), written (YES)• Open book/note, take-home

• Advantages: ↓anxiety, ask more application Qs• Disadvantages: ↓ incentive to prepare, uncertainty of who does work

Test Characteristics

• Question types• Semi-objective

• short-answer• completion• mathematical

• Objective • t/f• Matching• multiple-choice• Classification

• Essay

Semi-objective Questions

• Short-answer, completion, mathematical

• When to use (factual & recall material)

• Weaknesses

• Construction Recommendations (p. 151)

• Scoring Recommendations (p. 152)

Objective Questions

• True/False, matching, multiple-choice

• When to use (M-C: MOST IDEAL)• FORM7 (B,E).doc• Pg. 160-3: M-C guidelines

• Construction Recommendations(p. 158-60)

• Scoring Recommendations (p. 163-4)

Figure 8.1

The difference between extrinsic and intrinsic ambiguity(A is correct)

A

B

C

AAB B

C

CDD

D

Too easy Extrinsicambiguity(weak Ss miss)

IntrinsicAmbiguity(all foils = appealing)

Cognitive Assessments I

• Explain one thing that you learned today to a classmate

Review: Cognitive Assessments I

• Test types• Mastery Achievement

• Table of Specifications• Identify content, assign cognitive demands, weight areas• Provides support for what type of validity?

• Questions Types• Semi-objective: short-answer, completion, mathematical• Objective: t/f, match, multiple-choice

• Which is desirable: intrinsic/extrinsic ambiguity

Essay Questions

• When to use (definitions, interpretations, comparisons)

• Weaknesses

• Scoring

• Objectivity

• Construction & Scoring recommendations (p. 167-9)

Characteristics of “Good” Tests

• Reliable

• Valid

• Average difficulty

• DiscriminateGotten correct by more knowledgeable studentsMissed by less knowledgeable students

• Time consuming to write

Quality of the Test

• Reliability • Role of error in an observed score• Error sources in written tests

• Inadequate sampling

• Examinee’s mental/physical condition

• Environmental conditions

• Guessing

• Changes in the field (dynamic variable being measured)

Quality of the Test

• Validity• CONTENT key for written tests• Is critical information assessed by a test?• T of Specs helps support validity

• Overall Test Quality• Based on individual item quality (steps 1-8, pg. 175-80)

Item Analysis

• Used to determine quality of individual test items

• Item DifficultyPercent answering correctly

• Item DiscriminationHow well the item "functions“Also how “valid” the item is based on the total test score criterion

Item Difficulty

100*nn

cc

LU

LUDifficulty

0 (nobody got right) – 100 (everybody got right)Goal=50%

This allows for max item discrimination

Item Discrimination

100*minn

cc

U

LUationDiscri

<20% & negative (poor) 20-40% (acceptable)Goal > 40%

+ discr: incr reliability -: decr reliability

Figure 8.4The relationship between item discrimination and difficulty

Moderate difficulty maximizes discrimination

Sources of Written Tests

• Professionally Constructed Tests (FitSmart, Ask-PE)

• Textbooks (McGee & Farrow, 1987)

• Periodicals, Theses, and Dissertations

Questionnaires

• Determine the objectives• Delimit the sample• Construct the questionnaire• Conduct a pilot study• Write a cover letter• Send the questionnaire• Follow-up with non-respondents• Analyze the results and prepare the report

Constructing Open-Ended Questions

• AdvantagesAllow for creative answersAllow for respondent to detail answersCan be used when possible categories are largeProbably better when complex questions are involved

• DisadvantagesAnalysis is difficult because of non-standard responsesRequire more respondent time to completeCan be ambiguousCan result in irrelevant data

Constructing Closed-Ended Questions

• AdvantagesEasy to codeResult in standard responsesUsually less ambiguousEase of response relates to increased response rate

• DisadvantagesFrustration if correct category is not presentRespondent may chose inappropriate categoryMay require many categories to get ALL responsesSubject to possible recording errors

Factors Affecting the Questionnaire Response

• Cover LetterBe brief and informative

• Ease of ReturnYou DO want it back!

• Neatness and LengthBe professional and brief

• InducementsMoney and flattery

• Timing and DeadlinesTime of year and sufficient time to complete

• Follow-upAt least once (2 about the best response rate you will get)

The BIG Issues in Questionnaire Development

• ReliabilityConsistency of measurementStability reliability: 2-4 wks between

administrations

• ValidityTruthfulness of responseGood items, expert reviewed, pilot testing,

confidentiality/anonymity

• Representativeness of the sampleTo whom can you generalize?

Cognitive Assessments II

Ask for clarity on something that challenged you today