DOCUMENT RESUME

ED 194 613 TM BOO 744

AUTHOR Brossell, Gordon: Hoetker, JamesTITLE Training Manual for the Writing Subtest of the

Florida Teacher Competency Examination: Proceduresfor Training Raters and Referees in HolisticEvaluation of Essays Written for the Florida TeacherCompetency Examination; Volume Two of Five.

INSTITUTION Florida State Univ., Tallahassee. Coil. ofEducation.

SPONS AGENCY Florida State Dept. of Education, Tallahassee;PUB DATE Jun 79NOTE 73p.: For related documents, see TM BOO 743, TM 80(

745=47.

EDRS PRICE MF01/PC03 PIUS Postage.DESCRIPTORS Competency Based_Teacher Education: _Elementary

Secondary Education: *Essay Tests; *EvaluationCriteria: Higher_Education: *Minimum CompetencyTesting: Rating Scales: Scoring: TeaCherCertification: Trainees: Trainers; *Training Method8:Writing (Composition): *Writing Skills

IDENTIFIERS Florida: *Florida Teacher Competency Examination:__*Holistic Evaluation: Interrater Reliability; WritingEvaluation

ABSTRACTThis manual is designed as -a guide for the initial

training of raters and referees in holistic evaluation of writingsamples. it is intended for use by a single trainer in a group oftwelve (nine raters, three referees) trainees. The manual is dividedinto sections- corresponding to the procedures to be used in thetraining sessionsThe left hand Pages contain the trainer's text,and- include general directions and special points of interest. Theright hand pages_contain_reprints_from the Trainee Handbook. Basicinformation about holistic evaluation cf writing samples is providedin the forM_Cf_gUeStiona and answers. Ratings of exemplary essays areincluded. (Author/GK)

***********************************************************************Reproductions supplied by EDRS are the best that can be made

from the original document.. ***********************************************************************

-ar

TRAINING MANUAL FOR THE WRITING SUBTEST

OF THE FLORIDA TEACHER COMPETENCY EXAMINATION

Procedures for Training Raters and Refereesin Holistic Evaluation of Essays Written

for The Florida Teacher Competency Examination

Gordon Brossell James Hoetker

College of EducationFlorida State University

VOLUME TWO OF FIVE

SCOPE OF INTEREST_NOTICE

The ERIC Facility_has_assianedthis document for proeeYito:

JCIn our judgement, this documentis-also of interest to the clearing-houses noted to the right. index-ing should reflect their specialpoints of view:

U S DEPARTMENT_ OF HEALTH.EDUCATION & WELFARENATIONALINS_TITUTE OF

EDUCATION

THIS DOCUMENT HAS BEEN RE Pao-DUCED EXACTLY -A5 RECEIVED FROMTHE PERSON OR ORGANIZATION ORIGIN-ATING LT_POINTS OF VIEW OR OPINIONSSTATED DO NOT NECESSARILY REPRE-SENT OFFICIAL NATIONAL INSTITUTE OFEDUCATION POSITION OR POL ICY

Under Contract to the Department of Teacher EducationFlorida State Department of Education

#790116

Department of Education

June, 1979

"PERMISSION TO REPRODUCE THISMATERI HAS BEEN GRANTED BY

K h Al

TO THE EDUCATIONAL RESOURCESINFORMATION CENTER (ERIC)."

2

TABLE OF CONTENTS Page

OVERVIEW

I. Explaining the Process of Holistic Evaluation 1

II. Introducing the Rating Criteria 3

III. Rating the Exemplary Essays 9

IV. Determining the Effectiveness of the Training 43

V. Concluding the Training Session 67

Appendix: Trainee Handbook

OVERVIEW

Purpose. This manual has been designed as a guide for the initial

training of raters and referees in holistic evaluation of writing

samples. It is intended for use by a single trainer in a group of

about twelve trainees (usually nine raters and three referees).

Format. The manual is divided into sections corresponding to the

specific procedures that will be used in each training session.

The left hand pages contain a text for the trainer and include both

general directions, in lower case type like this, and reminders

and special points of emphasis, IN UPPER CASE TYPE LIKE THIS. The

right hand pages contain reprints from the Trainee Handbook,

corresponding to whatever step the training procedure is being .

undertaken. In cases where no material from the Trainee Handbook

is being used, the righthand pages are blank.

Time Limits. Each training session should last approximately two

to two-and-a-half hours, depending on differences in groups. Trainers

are urged to adhere, within reason, to these limits, since the con-

centration required for successful training is likely to wane if

the session is unduly extended.

4

EXPLAINING THE PROCESS OF HOLISTIC EVALUATION

This section contains some basic information about holistic

evaluation of writing samples, in the form of questions and

answers.

PRESENT_THE INFORMATION TO THE TRAINEES IN THE ORDER IT IS WRITTENAND DISCUSS_IT_WITH_THEM, ANSWERING ANY QUESTIONS THEY MIGHT HAVE.DON'T FEEL OBLIGED_TO READ THE MATERIAL VERBATIM. PUT IT IN YOUROWN WORDS IF YOU WISH.

What is_ holistic_evaluatiqn?

Holistic evaluation is an established procedure for rating writing

samples. It has been used for many years by professional testing

agencies, such as the Educational Testing Service and the College

Entrance Examination Board.

How does it work?

Each reader, or rater, reads a writing sample once, obtainini an

impression of its overall quality. He or she then assigns a

numerical rating to the paper, based on his or her judgment of how

iwell it meets a particular set of rating criteria.

How is zt different from_analytical evaluation?

Analytical evaluation entails close attention to specific features

of a piece of writing, which takes a good deal of time and yields a

detailed assessment. Holistic evaluation is based on impressions of

a piece of writing as a whole, so it occurs more quickly and produces

a less detailed assessment.

Are there different versions of holistic evaluation?

Yes, there are several different versions, depending on the purposes

of the evaluation.

-2-

.Which one are we going to use and why?

We are going to be using that version of holistic evaluation known

as general impression marking because our task is, ultimately, to

rate a large number of essays according to their overall competence.

General impression marking is the best holistic procedure for

scoring a large number of essays written for a particular occasion--

in this case, the writing subtest of the Florida Teacher Competency

Exam.

Aren't general impressions bound_ to be subjective, and thereforeunreliable?

Yes, they are subjective, but when used in accordance with certain

training principles, they can be highly reliable.

THAT IS THE GOAL OF THE TRAINING SESSION. BY PRACTICING RATINGSAMPLE ESSAYS AND DISCUSSING RATINGS, THE GROUP WILL BE SEEKINGTO ESTABLISH A CONVERGENCE OF SUBJECTIVE JUDGMENTS AT A HIGH LEVELOF RELIABILITY.

THE KEY FACTOR IN ACHIEVING_HIGH RELIABILITY, OR CONSISTENCY, ISTRAINING IT IS VERY IMPORTANT, THEREFORE, THAT EVERYONE BE WILLINGTO WORK TOWARD THE GOAL OF ATTAINING A HIGH DEGREE OF CONSISTENCY INRATING ESSAYS. THIS ENTAILS FOLLOWING UNIFORM PROCEDURES, STRIVINGTO DEVELOP LIKE RESPONSES, AND APPLYING THE RATING CRITERIA RIGOROUSLYTO EACH ESSAY.

SOME ENGLISH TEACHERS MAY GIVE GREATER WEIGHT TO CERTAIN FAULTS ORVIRTUES IN WRITING THAN OTHER ENGLISH TEACHERS: SPELLING, PUNCTUA-TION, SENTENCE FRAGMENTS, PENMANSHIP, TOPIC SENTENCES, AND SO ON.FOR OUR PURPOSES, SUCH PERSONAL STANDARDS WILL BE INVALID AND MUSTBE PUT ASIDE IN FAVOR OF UNIFORM IMPLEMENTATION OF THE RATING CRITERIA.

ASK THE TRAINEES IF THEY UNDERSTAND THIS REQUIREMENT OF THE TRAININGAND IF THEY ARE WILLING TO ABIDE BY IT. DISCUSS THEIR REACTIONS TOIT, IF THERE ARE ANY.

What scoring system will we be using?

The scoring system for evaluating essays written on the Florida

Teacher Competency Examination is as follows: each essay will be

read independently by three raters, who will assign.it a rating of

1, 2, 3, or 4, depending on his or her judgment of its overall

6

competence. (We will look at what these ratings mean in a minute

when we discuss the rating criteria.) If one of the three ratings

differs from any other by two or more, or if the three ratings are

some combination of 1 1 2, the paper will be sent to a fourth rater,

or referee, who will read it and assign it a rating. ThiS rating

will then replace the most diSctepant one. (We will lbbk at this

procedure a bit more carefully later in the training session.)

The sum of the three ratings for each paper--either the three

Original ones, or two originals and a referee's 848titUte"i8 its

final score. The range of scores for each essay is thus 3 at the

low end to 12 at the top.

II. INTRODUCING THE RATING CRITERIA

PASS OUT THE TRAINEE HANDBOOKS. ASK THE TRAINEES NOT TO LOOK AHEADIN THEM SINCE THE TRAINING DEPENDS ON A SEQUENTIAL READING OF THEMATERIAL; IN THE.ORDER IT IS PRESENTED.

TRAINER'S NOTE: IT IS CRUCIAL THAT THE TRAINEES READ THE ANNOTA-TIONS TO THE EXEMPLARY ESSAYS ONLY AFTER THEY HAVE RATER THEESSAYSINDFPENDENTLY.

Have the trainees look at page 1 in their Handbooksthe

Criteria for Evaluation of Essays.

How were these criteria chosen?

The criteria were developed to accommodate three specific conditions

imposed by the writing test:

1) that they reflect those characteristics widely accepted as

indicative of good writing and can thus reasonably be expected to

appear in essays written by people seeking initial Flcrida certifi-

cation;

2) that they are amenable to being "translated" into operational

descriptions of levels of competence that can be used by raters as

-4--

CRITERIA FOR EVALUATION OF ESSAYS

1. Rhetorical Quality

1.1 Unity: An ordering and interdependence of partseproducing a single effect: cOmpleteness.

1.2 Focus: Concentration on the chosen topic.

1.3 Clarity: Lucidity of expression; ladk of atbiguityand 'distortion.

1.

1.4 Sufficiency: Appropriate depth and breadth of expressionto meet the writer's purposes and the demands of theparticular topic

2. Structural and Mechanical Quality

2.1 Organization: Consistent and coherent integration andconnection of parts;

2.2 Development: Appropriate and sufficient exposition ofideas; use of detail, examples, illustratibnS, comparisons,etc.

2.3 Paragraph_ and Sentence Structure: Appropriate fbrril,variety, logic, relatedness of and among structural units.

2.4 Syntax: APpropriate ordering of words to convey intendedmeaning.

Observance of Conventions in Writing

3.1 Usage: APPrOpriate use of languagefeatures:_ infledtions,tense, agreement, pronouns, modifiers, vocabulary, levelOf diSCOurSei etc.

3;2 Spelling, Capitalization, Punctuation: Consistent practiceof accepted forms.

directly as the criteria themselves;

3) that they take into account the writing subskills identified by

the Council on Teacher Education.

READ AND DISCUSS THE CRITERIA

Have the trainees look at page 2 in their Handbooks- -the Operational

Descriptions of Levels of Writing Competence.

How were these descriptions developed?

The descriptions are based on the rating criteria; that is, they

use the terms which are definer in the criteria. The descriptions

are an attempt to express as clearly and precisely as possible the

imprecsions a rater has in terns o_f the criteria when he or she reads

essays of varying quality. For our purposes, there are four levels

of quality, or competence, a rater must be concerned with. There

could be fewer, and there could be more. Four is the best number for

our task because it provides enough d-,grees of:distinction to be

both meaningful and manageable in assessing writing competence.

READ AND DISCUSS THE OPERATIONAL .01.-RIPTIONS.

=6

OPERATIONAL DESCRIPTIONS OF LEVELS OF WRITING COMPETENCE

2.

4 The essay is unified, sharply focussed, and distinctivelyeffective. It treats the topic clearly; completely, and in suit=able_depth and breadth. It- is clearly and fully organized4 and itdevelops ideas with consistent appropriateness and thoroughness._The_essay reveals an unquestionably firm command of .paragraph andsentence structure. Syntactically, it is smooth and often elegant.Usage -is uniformly sensible, accurate; and sure. There are veryfew, if any, errors in spelling, capitalization; and punctuation.

3. The_essay is focussed and unified, and it is clearly if notdistinctively written. It gives the topic an adequate though notalways thorough_treatment. The essay is well organized, and muchof the time _it develops ideas_ appropriately and sufficiently. Itshows a good grasp of paragraph and sentence structure, and itsusage is- generally accurate and- sensible: Syntactically, it isclear and reliable. There may be a few errors in spelling, capital=ization, and punctuation; but they are not serious.

2. The essay has -some degree of unity and- focus; but each couldbe_improved. It is reasonably clear; though not invariably soand it treats the topic with a marginal degree of sufficiency; Theessay reflects some concern for organization and for some development_of_ideas, but neither is necessarily consistent nor fullyrealized. The essay reveals some sense, if not -full command, ofparagraph and sentence structure. It is syntactically bland and,at times, awkward. Usage is generally accurate; if not consis-tently so. There_ arelosome errors in spelling, capitalization,and punctuation that detract from the esoay's effect if not fromits sense.

1. The essay lacks unity and focus. It is distorted and/orambiguous, and it fails to treat the topic in sufficient depth andbreadth. There is little or no discernible organization and onlyscant development of ideas, if any at all. The essay betrays onlysporadically a sense of paragraph and sentence structure, and itis syntactically slipshod. Usage is irregular and often question-able or wrong. There are serious errors in spelling, capitalization,and punctuation.

10

Have the trainees look at page 3 in their HandbookS--the grid

showing the association of the rating criteria and the essential

competency subskillS developed and mandated by the State. Note

that all the subskills are) in fact, accounted for in the rating

criteria and will be evaluated in the hOliStic rating process.

(In other subtests of the Florida Teacher Competency Rxamination,

subskills will be tested separately rather than in the context of

general performance.)

BRIEFLY READ AND DISCUSS THE GRID.

11

FIGURE 1. Now the Essential Competency

Subskills in Writing are Evaluated by a Criterion

Guided Holistic Rating Prncedure

ESSENTIAL COMPETENCIES: Demonstrate

the ability to write in a logical,

easily understood style with appro;

priate grammar and sentence structure,

A. Differentiate between formal and

informal written English.

12B. Use language appropriate to the

topic and reader,

C. Apply basiclechanics of writing.

D. 4ply appropriate sentence

structure,

E,_ Apply basic techniques for

organization,

F. Apply standard English usage.

RHETORICAL STRUCTURAL CONVENTIONAL

g

ug 44 0 0 0

._ 0) o g 1,4 uN pi _g

44 a 41 0 0 X gx tn .r.1 .11 g it.i U OS Q. sd

OAs .14 U til q tg . )4 g SO U

44 U u) 0 p crl u) 2rl N t4) t? 01 N NI vt rl NI 1 4 I I I 4 Iri ri N N I tl)

dId

11.4444444404.44marrmwrotamomm

-9-

III. RATING THE EXEMPLARY ESSAYS

The group is now ready to undertake the central step in the training

process--rating sample essays holistically.

TELL THE TRAINEES THAT THEY ARE GOING TO PRACTICE RATING ESSAYS THATHAVE BEEN SELECTED AS GOOD EXAMPLES OF VARIOUS LEVELS OF COMPETENCEIN WRITING. THE ESSAYS WERE WRITTEN BY UNDERGRADUATE EDUCATIONMAJORS UNDER TEST CONDITIONS SIMILAR TO THOSE THAT WILL BE USED ONTHE FLORIDA TEACHER COMPETENCY EXAMINATIbN. THEY HAVE BEEN RATEDBY EXPERIENCED RATERS AND ANNOTATED IN ACCORDANCE WITH THE OPERATION-AL DESCRIPTIONS. AMONG THEM ARE EXAMPLES OF INCOMPETENT, AVERAGE,ABOVE-AVERAGE} AND EXCELLENT ESSAYS (ESSAYS RATED 1, 2, 3, AND 4)AS WELL AS ESSAYS WHICH STRADDLE THE BORDERLINE BETWilEN TWO LEVELSOF COMPETENCE.

EXPLAIN THAT THE GROUP WILL BE READING, RATING, AND DISCUSSING ANUMBER OF ESSAYS--9 OR 12 DEPENDING ON THEIR PROGRESS AS RATERSDURING THE TRAINING SESSION. FOR THE FIRST THREE ESSAYS, THEFOLLOWING PROCEDURE WILL BE USED:

EACH TRAINEE WILL READ AN ESSAY, ASSIGN IT A RATING, AND THEN SHAREHIS OR HER RATING AND THE REASONS FOR ASSIGNING IT WITH THE GROUP.AFTER THE DISCUSSION, THE TRAINER WILL PRESENT THE RATINGS GIVEN BYTHE EXPERIENCED RATERS AND WILL READ THE ANNOTATIONS ALOUD TO THEGROUP SO THAT THE TRAINEES -MAY COMPARE THEIR IMPRESSIONS WITH THETRAINING STANDARD. FURTHER DISCUSSION CAN THEN ENSUE IF NECESSARY.

Have the trainees look at page 4 in their Handbooks--ExemplaryEssay 1.

ASK THE TRAINEES TO READ THE ESSAY ONCE AND ASSIGN IT A RATING OF1, 2, 3, OR 4, MAKING SURE THAT THEY REFER TO THE OPERATIONALDESCRIPTIONS. MAKE SURE THAT EACH TRAINEE DOES THE RATING IN-DEPZINDENTLY.

When the trainees have rated the first essay, ask each to report his

or her rating together with the reasons for assigning it. Write

each rating on .the chalkboard as it is given. After the trainees

have reported their ratings and discussed them in the group, report

the ratings of the experienced raters and read aloud the annotation

to Exemplary Essay I. Call attention to the deviations among the

ratings posted on the chalkboard. (In the extremely remote

possibility that there are few or even no significant deviations,

(CONTINUED on page 18)

.14

-iU- 4.ESSAY NUMBER: I

TOPIC: What are the essential characteristics of a good teacher?

20dcW .ezze6u1

-4(44a) a A 4vz

ZXe) 42164;n4;c1a246=4,4.reek44. .ac46

X3weit- Wa4, Art4d.264EA;e .zzAaZeze

t&,

-4ze4, go7.xfAh6t,&z, 4eZ4ch

,z%z6t

alede:w.

dtzALek.A!,,,-,27.6e4Y a4-0- 4d.4.9a,

ole:of

eeteeZ-4 eg,Le

.2Ze,AzZei

e- -1-z;i-e404)

4e-Ze)r

15

-12- 6 .

1 7

-13=ESSAY NUMBER, 2

7.

TOPIC, Which person in public life do you most admire and why?

zzetz.

,e4,6144214e,14-/Lz66q .z446-2X EGG 0444Z444-

-00cz.erct.)

oc-t o.)e2 W.42z4L*.

tod -27).yte44,1,i4,

. ,

ESSAY NUMBERi 3

TOPICS Should sex education be taught in American public schoolS?Why or why not?

-15- 9.

-

75._!644-CALe

U/ze

-Ad& <5t-etz

-17- 40.

ANNOTATIONS TO THE EXEMPLARY ESSAYS

Exemplary Essay #1--An essay rated 2 by each of three experiencedratersAn essay of average competence

Topic: What are the essential CharacteriStiet of a goodteacher?

Commentary: This essay treats_the topic, thOUgh rather super-ficially; It reflects some concern for organiza-tion and development of idoati though in a rudimen-tary way._ Sentences are structurally repetitious;syntactically, the essay i8_ flat. Usage isgenerally accurate, though -there are obvious lapseson occasion, particularly in- pronoun referencesand agreement. There are a few minor errors inspelling and punctuation.

Exemplary Essay #2--An essay rated 2 by two expetionced raters and1 by a third experienced. raterAn essay straddling the borderline betweencompetence and incompetence

Topic: Which person in public life do you most admire and why?

Commentary: This -essay deals with the topic, but its_Styleimpedes its sense; The essay lacks clarity. Itis wordy and often syntactically awkward, prObig!mswhich are compounded by cumbert'omeI inflated_usage. There are some errors in spelling andpunctuation.

Exemplary Essay #3--An essay rated 1 by each of three experiencedratersAn incompetent essay

Topic: Should sex education be taught in American publicsehools? Why or why not?

Commentary: This essay does little more than state, repeatedly,the writer's opinion that there should be sexeducation in the schools. The topic is giveninsufficient treatment; there is virtually noevidence cited to support the position and nodevelopment of any related idea(s). The writeruses paragraphs and sentences, but they areconsistently ill-conceived. The paper is repletewith discordant usage, and there are numerousmisspellings.

-18-

mention thiS to the group--you may justifiably express surprise

in doing so--and go on to the second essay.) Have the trainees

read, rate, and discuss Exemplary Essays 2 and 3, individually,

in the same manner: discuss the trainees' ratings as they are given--

posting them on the chalkboard--report the experienced raters'

ratings, and yead the annotations aloud so the trainees can compare

their impressions with the training standard.

TRAINER'S NOTE: DURING THE DISCUSSION OF THE FIRST THREE EXEMPLARYESSAYS; EMPHASIZE THE IMPORTANCE OF MAINTAINING A CONSISTENT APPROACHTO THE RATING OF ALL ESSAYS, USING THE RATING CRITERIA AND THEOPERATIONAL DESCRIPTIONS. REMIND THE TRAINEES THAT STRONGLY HELDPERSONAL DISPOSITIONS TOWARD PARTICULAR WRITING FEATURES, SUCH ASEXCESSIVE CONCERN FOR SPELLING, SENTENCE FRAGMENTS, OR PENMANSHIP,WILL BIAS ONE'S HOLISTIC IMPRESSION OF AN ESSAY.

POINT OUT TOO THAT ESSAYS CONTAINING NONSTANDARD DIALECT FEATURESAND/OR EXTREME STATEMENTS OF A SOCIAL, POLITICAL, OR PERSONAL NATUREMUST BE EVALUATED--AS ALL OTHER ESSAYS--ACCORDING TO THE CRITERIA,REGARDLESS OF THE RATER'S PERSONAL FEELINGS FOR OR AGAINST SUCHFEATURES_AND SUCH STATEMENTS. TRAINEES WHO CANNOT OR ARE NOTWILLING TO RATE ESSAYS IMPARTIALLY WILL NEED TO BE EXCUSED FROMFURTHER PARTICIPATION IN THE TESTING PROGRAM.

The group is now ready for the second set of Exemplary Essays --

4, 5, and 6 (pages 11-20 in the Trainee Handbook). If the trainer

feels that additional whole-group discussion of ratings is desirable,

the essays may be handled in the same manner as the first three.

If however, the trainer feels that the trainees have made progress in

understanding and using the holistic rating process, a small-group

approach can be adopted, as follows:

Ask the trainees to rate the three essays independently, making sure

to reread the operational descriptions of the four levels of writing

competence before reading and rating the essays. Remind them to

read each essay only once. After everyone has read and rated the

essays, break the trainees up into groups of three. (If the total

(CONTINUED on page 2923

ESSAY NUMBERi 4

TOPICS

-19-

What values do you feel schools should attemptnts?-

to

11.

impart to

,okietra.i

iteiteoceit. ,-e&?.IeJ

ee21644.e .zzL

fie

62i

.e.p-ofzedee

/1.c...-Aele_-,--#...G.E _d__.1.41/-a_.? _.Z.-..___L-e-._, :L.-/ /

-..w, ... 2 _Ac&[!'...._ ,, 21_.. _- .

...."--Ir ...a..........._ _ --AT- .. I 4 - 4".. //_ A Ar''4 'e ''l ...1 r 2I.Lde..........9_1( ..! _...L.!_ ____..r -'"

_... -Alere 1 ..."..

......rAl _,.._,_;_._-rt -1 41111' ' L

l /Jr -._ -.. -./. LII-..... -/IlL , 2.4.......6,iiii. _4/Lmmer .' AY" ' .d ._ -., '

/ 1 ,

AL--.-741.arZimir.r.-4...............t--. A_ __,__Ai_.04

i.14../..fiL .0" .. ____,...,

I

/ At /-/-!/ _X/ .1._.:_-....C....-

i r 1, / ' / . `mot..f A? '.'//

.... f.I

di..22,Lic

410-' I<

4.11r1 leI'

II Af'41... ''#_....Z.rd

X, or

/e ,e' ..^ / I#1 A# : r I / iZ../ Air, / Ar ., , '// , . r/ e / .. ems, - t ,

/ ,..../

IIIIIIr-.....r AA -AL _AL-.../ / /

4.0C//I / 1W .... j rI4_ , - . # ie 4f di

i1te - _A7._. /1 oi 1 z1i -

4._ "'e ' ° "Jr.:title 41 d I I _. ___,Z - " ""_......_ _....6c-.6_1_,A.r_..., . ..4.,c_/ / 0/AO/ AP'

. Aiiel 1 i 19. _...a_ A A 00 /

to c7J

ru

-22-

4-e-02z-ecZ.frrLdez-,

a-eZ41-0-e4tJILezw4.

14.

27

"J--AAL-ez.1-1

_edie

-23-ESSAY NUMBER, 5

15.

TOPIC: What are the essential characteristics good teacher?

.45/-;e-LL6-Cct 4e

e2t4A-,46400-6),..441

x;)t)

e/-444-4E-e-ZfAiie4vc..Jz%.43e-

41

w//

e -

.- __/Al

/

_eird,

_Aor

_4f if,0119

v.et -ay.,'

AIN. -a

,

_ -

#1. __. #

'

/A4___A_-./41i!

A

_

/

-A-

<__

C

ESSAY NUMBER: 6

TOPIC: Which person in public life do you most admire and why?

-25-

1.

1,

,\

\\\\,,b,;i

tAi

A

,\1*43

.

, '',4_

1

\IA

\I\

IAA

i.

N'1

\\

\1N

444aa

11)''

11

\-1 .NN

'N'1'

A

I

sl1

1A

A

1

N.

`N

A \1

1

\'

`N\I

%

I

\\

NI

:1\1

1

\N

I\1

\ \\ \\

\ \\

NI

k\

N.

),,

1.

ki

I1

\,.11

1

kN

A '

1A

/

V**

\ \\

N.1

.1

.,N

I1

N N

1\

)X

\1

1

%1/4

1.N

4

'1

.11

Nvi

1

1

411

\

1

1

1

\\

.11

\SII

\\

1SI

. 41141)1,

11

11

I"

\\-V

1

,,

1kt

1N

\s*k%

N\ 1

N,±

.,11'11

11

i'1

1

Ok4:1

.411'11

)\

-

1/41

i\

\\

'\)\ 1

\41,1

\4/1

.'1

\\\

0\1-1

-29-

number of trainees does not allow all groups to have three members,

use groups of three and four alternately).

TRAINER'S NOTE: THREE IS THE SMALLEST NUMBER OP RATERS PER GROUPFOUND TO YIELD SUFFICIENTLY HIGH LEVELS OF RELIABILITY. GROUPSOF THIS SIZE_WILL BE USED TO RATE ESSAYS WRITTEN ON THE FLORIDATEACHER COMPETENCY EXAMINATION SUBTEST IN WRITING. HENCE THATNUMBER IS BEING USED IN THE TRAINING SESSIONS.

Have each group compare and discuss the ratings each member has given

each of the three essays. Appoint, or have each group elect, a

spokesperson, who will be responsible for recording the substance

of the group's discussions and for reporting it at the general

session to follow.

REMIND THE TRAINEES TO KEEP REFERRING TO THE OPERATIONAL DESCRIPTIONSIN THEIR SMALL-GROUP DISCUSSIONS. CHECK PERIODICALLY THE DISCUSSIONIN EACH GROUP TO MAKE SURE THIS IS BEING DONE.

After the groups have discussed the three essays, reconvene the

trainees in general session. Ask the spokesperson in each group to

report the members' ratings for each of the three essays, and list

these ratings on the chalkboard by group. Then have each spokes-

person report on the discussions that occurred in his or her group,

noting any special problems it encountered.

TRAINER'S NOTE: THERE WILL TYPICALLY BE MUCH MORE DISAGREEMENTABOUT AND DISCUSSION OF ESSAYS OF MIDLEVEL QUALITY--THOSE RATED2 AND 3--THAN OF ESSAYS OF LOW OR HIGH QUALITYTHOSE RATED 1AND 4.. IT IS ESPECIALLY IMPORTANT TO DISCUSS THOROUGHLY THOSE ESSAYSSTRADDLING THE LINE BETWEEN COMPETENCE AND INCOMPETENCE--BETWEENRATINGS OF 1 AND 2.

If the need arises, reiterate and discuss again the importance of

adhering to the established evaluation procedures--particularly in

essays containing controversial features.

After these three essays have been rated and discussed, refer to their

annotations.

-30--

READ ALOUD AND DISCUSS THE ANNOTATIONS FOR EACH ESSAY, NOTING HOWTHE ESSAY FITS THE DESCRIPTION OF THE PARTICULAR LEVEL(S) OF COMPETENCEIT HAS BEEN ASSIGNED BY THE EXPERIENCED RATERS. COMPARE SPECIFICFEATURES OF ESSAYS TO BRING OUT DIFFERENCES IN RATINGS.

Tell the trainees that when the actual writing subtest is administered,

there Will be two rUieS for referring essays receiving discrepant

ratings to a referee, whose rating will replace the most discrepant

one:

1) if any rating in the set of three differs from any other by

two or more; and

2) if the coMbinatiot of ratings is i 1

List all possible combinations of ratings on the chalkboard and

underline those falling under rule #1; check that one which falls

under rule #2, as follows:

1 1 1 1 3 3

1 1 2V 1 3 4

I 3 1 4_4_

1_1_ A 2 2 2

I 2 2 2 2 3

1 2 3 2 2 4

1 2 4 2 3 3

2 3 4

2 4 4

3 3 3

3 3 4

3 4 4

4 4 4

Discuss the rules briefly; if necessary. Call attention to the

trainee ratings, listed by group on the chalkboard, of the three

Exemplary Essays just rated and discussed. Point out the ones in

each group which are discrepant.

TAKE A SHORT BREAK

35

-31-

Exemplary Essay #4--An essay rated )4 by each of three experiencedratersAn excellent essay

Topic: What values do you feel schools should attempt toimpart to students?

Commentary: This essay is sharply focussed, unified, anddistinctively effective; It treats the topic_clearly, completely, and in suitable depth andbreadth; The essay is very well organized anddevelops ideas appropriately It has a firmcommand of paragraph and sentence structure, andit is syntactically smooth: and, at times, elegant.Usage is uniformly sensible, accurate, and sure.There are no errors in spelling, capitalization,and punctuation;

Exemplary ESSay #5--An essay rated 2_by two experienced raters and3 by a third experienced raterAn essay straddling the borderline betweenaverage and above-average competence

Topic: What are the essential characteristics of a good teacher?

Commentary: This essay treats the topic adequately though notin much depth. It is well organized: it is, infact, a classic, schooI-taught, five-paragraphtheme, with an introduction, a body, and a con-clusion. Syntactically, it is flat and repetitiousUsage is pedestrian. The paper contain3 few, ifany, errors in spelling, punctuation, and capital-ization.

Exemplary Essay #6--An essay rated 3 by each of three experiencedratersAn essay of above-average competence

Topic: Which person in public life do you most admire and why?

Commentary: This essay is_ well focussed and unified, and it isclearly though not distinctively written. It iswell organized, develops ideas appropriately, andgives the topic suffiCient treatment. Usage isgenerally accurate, and syntax is reliable; Theessay shows a goOd grasp of sentence and paragraphstructure. There are a few minor errors inpunctuation.

36

-32-

Reconvene the trainees and have them look at pages 21-29 in their

Handbooks--Exemplary Essays 7, 8, and 9. Use the small-group

approach described on page 29 in reading, rating; and discussing

these essays.

TRAINER'S NOTE: IF THE:SMALL-GROUP METHOD WAS USED FOR EXEMPLARYESSAYS 4, 5;. AND 64 iASK THE TRAINEES TO FORM NEW GROUPS OF THREE INWORKING WITH_THESE1ESSAYS, SO THAT EVERYONE MAY SHARE IMPRESSIONSAND RATINGS WITH AS MANY DIFFERENT RATERS AS POSSIBLE.

After these essays have been read, rated, discussed in small groups,

reported upon in general session, and posted on the chalkboard,

refer the trainees to pages 40-43 in their Handbooks--the Annotations

to the Exemplary Essays. Read the annotations to Es8ays 7, 8, and

9 aloud as the trainees follow along, and discuss them with the group

in light of the ratings given by the trainees.

TRAINER'S NOTE: THE ANNOTATIONS FOR ALL TWELVE EXEMPLARY ESSAYSAPPEAR -IN -THE- TRAINEE HANDBOOK, THOUGH UNTIL NOW THE TRAINEES HAVENOT BEEN DIRECTED TO LOOK AT THEM. _ _!!__ NOT TO LOOKAT THE ANNOTATIONS TO ESSAYS_ 10, 11,_AND 12,__I_F_THEY ARE TO BE USEDIN THE TRAINING. SEE SECTION IV, BELOW.

(CONTINUED on page 43)

37

-33-

. , ESSAY NUMBER: 7

TOPIC: Which p on in blic life u most admire an

21.

--rx714-d..A.. .ee-dit.e.ace..t.

tkL4-

/etF-14..t.eAl, C2ft.-

,14-;z_az..c A -74-

zn,a2e.:07t., rA

Itaz ..-cidutot,x-1, rAt.4,1-4.t

.*tio

/77-gt,) 74Ze,9't

C\1

cV

I4

4,

\'\,JV

I

,t

.,

i.

1.4

tl\

1\'

I

1'1

\)1

\\

11\

I1\41

):

4"i'

9

\j

I

1 \I,

\I

11.1e

\NI

,.411

:si

\1

1

49

NO

11

1I

N;41

\y

'k1

1\

111

NIt

vl

1.i

ei

'.

\I1

%

,\

\'

1\I

II4

\'N

.

)N

,1:i

N,

)'

,v

1

%i

$1

N1

'NI

'',j \''si

11

".

\,

N;1

'''

., .i'tJ

iJ

,)

\-1

41.i

,1

,\11

ii

it*

Nk

411

1)

1,1

1

-%

)1

1I

.4

)

\

,I,

),

I

-41

.1

)s;

41

s)

hli1

%ti

41

1,:i

\ N1

1,1,\

\il

)'

i;

i',,,

:,)

i1

%1

i

\:*e

,,il

.1

g'l

io1i\

N1

1)1

N

NI

I\N

i

\

1ibk

\

14

NI

or

'1

I

Vyy

r

-39-

ESSAY NUMBEas 9 -4U-

TOPIC' Haw much and in what ways does television affect you?

0-424;42-,-A) ..e.ettALWeJ2 .<;zaketx.) ze.04

2/1.40

28.

216.44oG).i). glatdx. tott4tVci

.4egaieti.44.0,4P .4zitzo'

'14444141Yole.ineizi.

ryy

..e..zeL,ilt.0.0J,a.zoted.e6/6.

4.1656%C ./ta# (224tee.= etee-s4-I./3Pt 4cd.C..g4t) -4V-ZA:CIAJ r4623

eie" ?ife-vAee')c) -114:1c)zg/PLW Ott) AZIOZO

ez4c.e.

41.000111use)

29.-41-

. e&t.) eiLe;,JtJ.i; 6Wit/g;GAL*1 Are .4drelf ./Abgtd.

. ..Zekca

...404a) G&%7tet

./&6417L)

a/ttL '&24t.4.).41 le eiooc;

AficeziAtt/

iiitt(OZ.4e ec'T.ILT.A.JC.0

11-adoi, ex-0 0-zt.t-',4.0.4.4-,

otwaA.1,zevbbip.A

-42-

Exemplary Essay #7--An essay rated 1 by each of three experiencedratersAn incompetent essay

Topiq: Which person in public life do you most admire and why?

Commentary: This_ essay laCks_unity and focus. It wanders offthe topic and -fails to develop at all the two ideaspresented in its opening lines. Paragraphing isvirtually nonexistent, -and there are numerouswrongly used sentence fragments and faulty con-structions. The syntax is often muddled. Usageproblems abound: inappropriate abbreviations,infleCtions, wrong words and wrong fortS of WOrdS.There are many errors in spelling and capitaIiza-tiOn.

Exemplary Essay #d--An_ essay rated 3 by each of three experien-cdratersAn essay of above average competence

Topic: What are the essential characteristics of a good teacher?

Commentary: This essay is reasonably well focussed and unified.The_writing is generally clear. The essay i8_well organized and treats the topic sufficiently.Usage is usually reliable, and a good command_ ofsentence and paragraph structure is demonstrated.Syntactically, the essay is reliable if_a bit flat.There are a few minor errors in punctuation.

ExL-TioLary ssay #9--An essay rated 2 by two experienced raters Arid 1by a third experienced raterAn essay straddling the borderline betweencompetence and incompetence

Topic: How much, and in what ways, does television affect you?

Commentary:. Except for the second paragraph, this essay doesnot treat the topic; There is some developmentof ideas which are themselves only tangentiallyrelated to the topic at hand. Organization isspottyi though a sense of paragraph and sentencestructure is revealed; Usage is reasonablyaccurate; Syntax, except for an occasional lapse,is fairly consistent; Spelling, capitalization,and punctuation are good.

=43-

IV. DETERMINING THE EFFECTIVENESS OF THE TRAINING

At this point, the trainer must decide whether the trainees have

achieved an acceptably high level of rater-to-rater agreement in

their holistic evaluation of essay samples; If so, the training

session should be concluded according to the procedures outlined in

Section V. If not, Exemplary Essays 10, 11, and 12 should be used

for additional practice, using the small-group method and the

annotations in the same manner as for Essays 7, 8, and 9;

TRAINERS NOTE: ACCEPTABLE LEVELS OF INTER-RATER AGREEMENT MAY, INYOUR_JUDGMENT,A3E REACHED BEFORE_THE__RATING_OF EXEMPLABILESSAYS1=_9 HAS_OCCURRED===AFTER_THE_FIRST SIX; FOR- EXAMPLE. _IF_THIS_IS SO,USE ESSAYS 7, 8, AND 9 TO CONFIRM YOUR JUDGMENT. THE TRAININGSESSION SHOULD NOT BE CONCLUDED BEFORE THE FIRST NINE EXEMPLARYESSAYS HAVE BEEN READ, RATED, DISCUSSED, AND COMPARED WITH THEIRANNOTATIONS.

(CONTINUED on page 55)

Ag

ESSAY NUMBER' _10'TOPIC' What-Val

st

-44- 30.

you feel schools chould attempt to impart to

062--d-Ltz-Z, C-0-7t.tc

cLtA;Ictiz--

074Z-e

it& wz,.J

.RJ

6L, eeg,c,:t-tete.

iL'

414-=x-el.

49

71e=et

)4t-fr

V,e(..t

tMIzrIts

Nn

4 7 -

t c;744.0,1/4

4.t.t4.2td

ciA 2

ig4z 41-e41.t

v..-22 at .

L..1

a-Ctr7L--

33.

...t..e4*

G1.

-48- 34.ESSAY NUMBER: 11

TOPIC; Should sex education be taught in American public schools?Why or why not?

ea_Ater...)

-0-440-4sed-/.4iJ

ti/t.42,0JeLec..4eZr...e,,ae,t,.Ziageo

eX;R.74etzecd

.4_40.4.5-2ead;

6-..:z--1-4.--Ze-x-)

ta-)

424-!et

eak7..,44outett

ee;Lt. 7,444AJA.-4_tA411a,4 ,&et2t

-e&) 2k6oe(-en oe} .12te-,tt

_Zee.

.moo -

n

av

-50-

..1:2-467-aoe A__

30.

ESSAX NUMBERS

TOPIC*

12-51

Do_-ou favor -o; oppose the WOrrieExpl n why;_Uni d State

s liberation movem

a/Vr%92

d,er2._1

.4444.

W,A121-7*..42.4.442-.

.Z.-0112

&t-

da.47 n2A,E 6LA d444_

i44z44-ett4af

4e,

5 6

V1

,

1N

I

'

OS

kJI

\1

N

iN

i\

)...

'Nk'

N 1

1.1

41

11

\1

\1N

sI \1

4k

N,

\N-1 A

1

.:,

\1

\1

j.41

No \,

Cilik ,

11Nt44

N

NN

1

NS

.',

0''.1,1

\''

IN

ob

a\

''j.

.1

1\

\s \ \ I1

N

11

1

1)

1I

41

I

1N

*]1

11

411

11

%

IA I

1

1N ),

1.

NZ

NI

,..

1)

1

\\

k1

I\I

'

\1

,sq

1%

vi.1

1t1

1,1..

1

:1.

N.,

,.

\11

Aski

.11

IN11

iI

I1

\ I

t

6

lb

Qo

-54- 13.

Exemplary Essay /110 - -An essay rated 3 by two experienced raters and4 by a third experienced raterAn essay straddling the borderline betweenabove-average competence and e%cellance

Topic: What values do you feel schools should atteMPt toimpart to students?

Commentary: This essay is unified e d well focussed: It isclearly and, at times, distinctively Written. Ittreats the topic with considerable thoroughness --given the writer's poi:.t of view-. -and It isorganized surely and effectively, developing ideassufficiently and appropriately. The essay hasa firm command of sentence and paragraph structure.It is syntactically smooth. Usage is sound andalmost always accurate and sensible. There arevirtually no errors in spelling, capiValization,and punctuation.

Exemplary Essay #11--An essay rated 2 by two experienced raters and3 by a third experienced raterAn essay straddling the borderline betweenaverage and above-average competence

Topic: Should sex education be taught in American PUblicschools: Why or why not?

Commentary: This essay treats the topic adequatelY If notthoroughly. It gives some attention V° focussingand shaping and has some degree of uni-tY. Itcontains some sentence frasments, one (3-1' two ofwhich may be deliberate. There is some syntacticawkwardness, though much is clear. Usage isgenerally accurate, though some faulty Pronounreferences are present. There are a few errorsin spelling and punctuation, but tot Serious ones.

Exemplary Fssay #12--An essay rated 2 by each of three exPeriencedratersAn essay of average competence

Topic: Do you favor or oppose the women's liberati°n movementin the United States? Explain why.

Commentary: This essay is a reasonably clear and reasonablysufficient treatment of the topic; Ituffersat times from awkward syntax, faulty sentencestructure, and "overloaded" paragraphs! exceptfor an occasional lapse, usage is fairlY accurate.There are some errors in punctuation.

-55-

The trainer's decision rests on a personal judgment of the effects of

the training on the trainees' progress as raters; To guide his or

her judgment; the trainer can refer to the following expectations,

which experience has shown to issue from a well-managed training

session involving serious, co-operative trainees:

1) the ratings of essays done aftcr some practice and discussion

have taken place should show greater inter-rater agreement than those

done at the beginning of the training session;

2) as the training proceeds, discussion about ratings should center

more pointedly on specific criterial elements and their execution in

a particular essay;

3) raters should show an increasing willingness to use the operation-

al descriptions of writing competence in rating essays and to cite

relevant elements of them in discussion;

As an objective measure of progress; the trainer can keep a record

of the number of essays receiving discrepant ratings (those which

would be sent to a referee). The number of these should decline

as the session continues and the trainees become more accurate and

reliable raters. While it is diffitult to predict the actual number

of essays which will receive discrepant ratings in a training

session, it is not unusual for up to 50% of those rated during

first attempt at holistic evaluation to receive discrepant ratings.

Similarly, a drop in discrepant ratings to about 20% or 25% is not

-5b-

uncommon for essays rated near the end of a training session, after

considerable practice and discussion have taken place.

TRAINER'S NOTE: IN CONCLUDING A TRAINING SESSIUN, CALL TO THETRAINEES' ATTENTION AS MANY OF THESE INDICATORS AS HAVE BEEN APPARENT.

If the trainer feels that additional training is warranted beyond

that provided by the twelve Exemplary Essays, another rating session

can be conducted using the three backup essays included in the

Trainee Handbook (pages 57-66 in this manual).

IT IS EXPECTED THAT NO ADDITIONAL TRAINING WILL BE NECESSARY. THEBACKUP ESSAYS ARE PROVIDED MAINLY IN ANTICIPATION OF ANY SPECIFICPROBLEMS THAT, IN THE TRAINER'S OPINION; CAN BE ELIMINATED BY ONEMORE BRIEF RATING SESSION.

(CONTINUED on page 67)

TOPIC c;;57_ '4i

Should sex education be taught in American public schools?Why or why not?

4.11 400013 mcwai.,aAt. t.Atotuatil 4r tamklut. avall atw4 tixt4 4itt.

:icAttaa- ocitorapht .1)t. Igo iatAcazatlruis yawl _.tatiafat _tf

eau:L.44 _

----7 '40 tt-ott;ate.a.----dtki. 4itt-

_41/4ut

.iptaitst.= tivid64:414(02241.fajta4_.ktiwybecil_Adv

m -pbataL 40ixtutcchL.44tat... thitotiwt. oh dot. fkra&v.

.4..fvubciA.4(24.42.d.

atew._ gaili_hp,r 44-ku,t_thacaot cdatapxtuvo 2a.zet-Aivik,alaia ot.i444.1.4=4.

vit.taateut ut 4411 / toixh_ -tcteb _

-Pkt tat4_&Lnt abace_ma___

astItt.vt_6t kW OW Leal

Lo La. cioau (-4

.rwtat aptizuka..

6-Puick ctul. cr.6

Co

44-an c.'n fixtut.buo 41AtAt ad-u

otivetuat. usx.0 _Lk cokAlik pti,.4644Auk 6.0142-NOK4. _

ezbitt,t.-4- .41,g eary,z4

G2.

-58-

rncuu, J _hcadqt. (A6LLAtaeitaidam- JwiztAirbAr..4tkalAz41,_____ _

nti-rwtaigtt._-k) -1-ht_inal_4tipm 0.6

ractia,- Gott:ckt

--;: tata-4.4.1Atcaut__44xL...ku,th__culdit___e_ctr. _Acute__

6/1.-um44.47 -a-

-40 cutt. 44ki

J2,Clukcal.a. WaCcd.dA4Ot.-vA_--PfuAj. -Thc4

lor(luAl 00 GOegitkuat_ Les . Otikoist4.

4Cutki Cit.&41-ktA-4, ttacii.e6 culd.

CL tYLLA. akadt24 GULOttLgifa."4,,

1146 _ AKtithict.

vim.0,04,x(2.

_121. taell. 4

_it.e/ht _

ca CC a 64-a4 ,Ct

fra.41.ittt _akatAA

t.ri Ct4, CutAt. ,

1 9..

4 fFAMX.1)? CLrae... ()C. thicum

rA-11A-t- (-0.-Lpttirakti 4b1/14-1yvaA

..e.ttiyukt,t2 fv-ooi 04.

1\4. Joiapt

unk.pLuw-r, 6.4.63

-59-

vaiioul 4utia.k..6.1 .611.44-0 cuttot itic4-he

cuttolzt pfmcveht,tos 4-hcLr. C/a

ILAIW irtA941t slAktOtaia.

cut&. ctiYaLa

413 t ,4 AAAL Jute.d.catzt_wauut___Atit PrI4Aft__+0 JAtiat.pr.tft_

4110 haxl ,________LIOutria

t---

ccrupt cal_ al%-u _ sturyttqaut4AA _ _

00 -Th.t.01.__atiaavo.__ Oitaecti1 e. 4-#LL A-V4 #11.41-1_

tonl-tet. 4hd.t. 6.6thaitco amt at644-tuit4 C ashadattail-et.___MCUVLIA

-414teAt _fru-Liak -Oitow-r3_4ht.hicakt _ /num ittit

TOPIC: Do you favor or oppose the women's liberation movement inthe United States? Explain why.

wane& ut_ -tfik. cadCant A _

_Optn/Ln-) thpz0,5(6t. WoAcis

lien_1&_ AT,

47.

- - -- ^

meribtat)I min I V

_LrLST-cufS

)ham l S. _ _

2.1-1 terteA_ OtA,14

I

af,- tAXtm,--cashon -=thIctt- tai 4g br

If) tirte_ 0 V\ 6uCuLt Ltd(t

have taian arotwo hoit[Q_. -taAe.r)

Views,

rnirvis -mac _z)r\L0DrY\exs

P6 leru,1

-fricaL IS _ bOhoLl'.

rno\feriAted--ir\ OrNj

AettAti 0 aAaA__1.41A ,t

_, . 0 _IA A AA1 el' .0

,P A. tip

0 Al' At. 0 0 a 01 Ato 0Pet

au ' ili' ',A 4°4 --/__el 4, / 0"0 A _ el ° 1 oft._i 0 4 0, Aiw o At ' 0, a. A I 4 i 2,011.1A A i #

r r

114 -1..: .,z.----.---- ,ZAA2 ifeeti,,t1

11111 il I g........_-,..

/ i f/i.-...e.c.,.....-____..a.

41.

A0 0 ii Act, 'ILe,_,,, // /,r t I-/ sk i /AM, __A

F,A# 47 I t irl,, . __.4., O a./ 0 If

k 0 dedj

AO 4 _ftt_.&Ad or , :_ez/ A 01

r / r i A -A-ie j0 1 100. AL-..vale" A.

/ i 4f1 1 A _AP--A7

tz sof// if , / _

..L

Al J i01 1 il I -_A_..

"td......4........2= 0 Irl

4 / /

1 -

"IOW 49Va::riiPO'orm-_-_ 1 w lirA72L' g.e=arj.

I / / 0 /

J I/ I 4 if

-L-Adia 0 1

1 / i _AV/ o!ILEk 4 / / ' 1 -MA_

,Or Ii

._dJi ' A I® 0 " - i i .4 4ft i '_atia

A4 i 4.A10 I 0 , 0 ,fp&_' / I t0A, , ../..f 0/,,Ij IL-4 124 4 , -0,90 A A gratUet

4.1 OfJ A Ax_ .L if wif

_._...&_ _I Air 40 /40Aiyal/---

ir: at -t AIL

te' t_. 41

VAr,, _Ai_i_i p

./

11Al -.LA 0 / 1

%._ i if 4_1! /

4 I'i Al i AgUi_ ....,Ali1 / 4 4,0,

_dAt_.1.0 I 0 ,d 1 0 0 tit faticii.da...,,

VI ala.*

4 ik_ _LIA 0 ,a at* 4 0 L,_

it / 0 0, 0 / 1 A.Itti ...yI

AZi_ 0- / 1 AFL 04A 0 0 0 a

) 0 0 A !SAL, ,1_AV dir

_ .. 'J 40 1 I 0 4 0 d / i /L_44 04 Iiii

oA ...._

44AL, &LA' S gf 0fil it*, ciL 01,4

1 A0 0°411 '_ _dAsa

-63-

c_itigectra_

me-t- tkx,

-eitPymote,_//

trIca lat-64)

5

-64=

TOP' Ct *Idol% 1,orson in public life do you most admire and why?

- _ Avila _fvu.4Uattit.t.cds

_ _..CAAtrut, L.124..a.ce te4__ _manEZird..? ryt64.-{_,ALut,tca.

nal.C.44.3 Gtmota__ait.d_ atak44_,C.4.2./.4_

.60 vittA)_..AAA1

1!

_ _ _ .Gtwii424_%.1,h440 _Mar% -fa

I cw a _ireci.n.4

.C4stct a4azrrtucaxia- cauaceve

_.Y4s14,1, bn 61444-inacLAZon-- io

_ifs)

aocir

__CWLCZfrarte.

EAU --4-ht 61-4142A

;_

&IA- Mott ku,r0.41.-

41 x-tt _ &ed. k6441- -40

ct . _ _a.rual a. w tat.044_ ct nuticat.9_.e4 .tutut &teal_ JUSK. LOCtLit.-.JFIAA4-he -

C4 ni-ek,gArl Jaz;v1.61a,tt,_ vta,aut _

kActe0. eApbct.4..044,,td-

JAA:O. Qt.)...n iatvzi .=-4Ct ft.ak _at

Li:11.th t..14 L. tc-nt-u:twz.d._.04L.-tG _4411. _

-06fvet"iodue4,_ Ataikr and _ Oft4,3aAJDL _

PAZ &tail cth tiu.act..1 a&Lot. -t-he

,r-rtirwit tAihytnwatt toctio nrwAA _

4=-heun

01224 m (AXIAICL.

69

AO

I A

4,1 e7

4 a

_1 11. e AL. j

4 4 It

_ L 4 s A A A r 4

4 .. ,,41 S's i 4 4.4.....-.._ if 4-44_ ....- -

A__., I 4' .3 IA . Li_ 1.4 O `4' 4

4

V F l

4 t

4

&A - 6 4

4 L it O

.4

4 rr

.4 I a

_...111111.

4.8..

I

2_4..2______2_

4

4

as. 4 IS = .A'-_ _a _...AL--.4 A__ . 41we.

44_ a A _. _Alf .4.,,t _e__A # J _a_____./ 0 a ./At!.._ 0 . I

.,1_ . __0_,...-4_14t a A.2...A.t 421..... a 0 0 0V

a_ . L__,, , _A' _a-ie.....1...AA.....a. A a .

I

e 0 4

I' :1...-_ .! o- kW__ 4. tf

.t 4 - air -

ht =66-, 5 3 .

ki a wadi,

uctiCtui-a-Azt- dPiAtdtioc... -ear124.

azzi _ With

haRAL___43!..trLtuatiit _ a_ttiltricit_p0weit _LOr:4_if; 4 A - Jt - cfal_Ciatithat_earftA 0 _.014M-144,1

_=Jra. ..1-4i0._ _ 6atittitile-e-.Idan_4./.___Ozt___ILL-mt caut.e___A4k=li- _--

..._. ateiztv.to-k

. sO6--Lz L.6 tu&v.,

ttA1... u%tluoCet.

he ..a. atuu -"Aux..

auA4ak44.4 ,Aitet

71

-67-

V. CONCLUDING THE TRAINING SESSION

Have the trainees look at page S4 in their HandbookS"the Guidelines

for Raters.

READ AND DISCUSS THE GUIDELINES. REMIND THE TRAINFES TO KEEP THEMHANDY AND TO REFER TO THEM JUST BEFORE RATING A SET OF ESSAYS.

Explain the schedule that the trainees will follow during the time

they are evaluating essays. Assign the trainees to the work stations

they will using during the evaluation sessions. Answer any final

questio;L; ghat arise. Dismiss the training session, giving the raters

a report to their stations to begin work.

-68-

GUIDELINES FOR RATERS

Before beginning to rate a set of essays, remember t

1. Reread the four operational deacriptions of levels ofcompetence.

54.

2. Establish a mental "set" so you are ready to concentrateon reading and rating the set of essays, withoul; distraction, from start to finish.

While reading a set of essays, remember to:

Read and rate the essays in the exact order you receivethem.

Approach each essay separately; don't compare one withanother.

Read each essay only once, unless the quality_ Of thepenmanship or waning concentration demands a rereading.

Avoid being influenced by your own dispositions towardparticular writing features (spelling,sentence fragments, topic sentences, penmanalaip, etc.).

). vold being unduly influenced by nonstandard_4ialect featAfeatureand by extreme statements of a social, political orpersonal nature.

6. Stop after every ten papers or_ so and take a short mentalbvjak; relax; take a deep breath, etc.) before returningto the rating task;

Referees' should make certain to:

1. Read the right essay.

2. Return the essay to the exact place in the set It comesfrom.

Record the rating next to the proper code number on theRater's Tally Sheet.