Evaluation, Testing and Assessment

June 9, 2011

Curriculum Evaluation

• Necessary to determine– How the program works– How successfully it works– Whether it meets learners’ needs– Whether more teacher training is needed– Whether learners are in fact learning

Purposes of Evaluations

• Program Accountability– Examines the effects of the program

• Program Development– Designed to improve quality of program

Types of Program Evaluation• Formative

– Focuses on ongoing development• Are placement tests accurately placing students• Is methodology used by teachers effective• Is the pace of material adequate

• Illuminative– Looks at how different aspects of program work

• What types of error correction strategies are used• How do teachers actually use their lesson plans• What are the teacher-student/student-student interaction patterns

• Summative– Determines the effectiveness of program

• Did program achieve goals• What did students learn• Were placement tests adequate

Who is involved in evaluation?• Insiders– Teachers—can provide formative evaluation of course– Students—can provide summative evaluation

regarding individual improvement and effective teaching

• Outsiders– Can provide objective perspectives

• Consultants• Administrators

Quantitative vs. Qualitative Evaluations

• Quantitative– Expressed numerically– Usually require large pool– Data can be analyzed and conclusions can be drawn– Format may be tests, checklists, surveys, self-ratings

• Qualitative– Subjective– Small pool– Collect information from natural settings – Format may be observations, interviews, case studies,

etc.

Testing

Tests—method of measuring a person’s ability, knowledge, or performance in a given domain– They occur at designated times• A subset of assessment

– Method—must be clear and structured• Multiple choice, writing prompt with rubric, oral

interview with checklist– Measure—from measure of individuals’ abilities an

idea of general competence can be made(Adapted from Brown, 2004)

Assessments

Assessments—an ongoing process that encompasses a wider domain– Occur throughout class period– Classroom activities are opportunities for

assessments but students should have opportunity to take chances and not be penalized

Informal vs. Formal Assessments

• Informal– Comments/Responses– Smiley faces– Notes in the margin

• Formal– Tests– Journals– Systematic observations

Formative vs. Summative Assessment

• Formative– Majority of assessments are formative– Informal– Focus on the continual development of learner’s

language• Summative– Designated time– Measures what students have learned over period of

time– Should still provide feedback

Norm-referenced vs. Criterion-referenced Tests

• Norm-referenced tests– Individual’s score is compared to a mean score– Quantitative results– Predetermined answers– SAT, NJASK, HSPA—most are not norm-referenced for

ELLs• Criterion-referenced tests– Individual gets feedback– Material is linked to curriculum– Result in grades

History of Testing• Discrete-point tests—each strand tested separately• Integrative tests—combining strands in testing but do

not provide overall performance ability– Cloze—activities with missing words– Dictation—writing what is heard

• Communicative tests—include grammatical, textual, illocutionary (speech act e.g. requesting, warning, etc.), and sociolinguistic components

• Performance-based assessment—usually interactive and less formal e.g. oral interview

• Multiple intelligence—move beyond simple IQ tests

Computer-Based Testing

CAT—Computer Adaptive Testing

Thoughts on computers grading essays??

5 Principles of Language Assessment

• Practicality• Reliability• Validity• Authenticity• Washback

Practicality

• Tests must be practical which means– Not too expensive– Not too time-consuming (in administering and

grading)– Easy to administer– Predetermined scoring

Reliability• Tests must be consistent and dependable• Factors that make a test unreliable include– Learner-related issues when a student has a bad day– Inter-rater issues when 2 or more scorers are not

consistent– Intra-rater issues when scorer is not consistent over

time– Test-administration issues when setting, time, and

other factors are not consistent– Test reliability issues when test is not clear or is too

long

Validity• Tests must produce results which are appropriate, meaningful, and

useful (Gronlund, 1998)– Content Validity

• Does test relate to material covered in class?– Criterion-related Validity

• Does test measure specific objectives?– Concurrent Validity

• Does the result on the test match to real world ability?– Predictive Validity

• Does the test properly place student at correct level?– Construct Validity

• Doest the test support theoretical constructs (theories, hypotheses, etc.)?– Face Validity

• Does this test appear to measure what it states it will measure (according to the students)?

Authenticity

• Test must have meaning in the real world– Tasks represent real-world tasks– Language is authentic (and not textbook language)– Material is contextualized• E.g. authentic materials such as newspaper articles,

signs, pictures, etc. are included as opposed to random sentences or fabricated materials

Washback

• Washback is the effect of testing on teaching and learning (Hughes, 2003)– Teaching to the test– Providing feedback on work so that student can

accurately achieve final goal (e.g. comments on rough drafts)

Washback helps develop learners self-confidence, interlanguage, language ego, and motivation.