Manual Data Collection

5 Manual data collection procedures

There is a long history of teacher supervisors recording information man-ually as they observe lessons, either by using open-ended note taking togenerate field notes or by using an observation system. There are advan-tages and disadvantages to both.

We will begin with a case to contextualize these issues and then exam-ine the use of field notes. We will discuss the pros and cons of note takingas a data collection procedure and learn some strategies for using maps,seating charts, pictorials, and sketches, as well as a shorthand systemto record information quickly. In the second part of the chapter we willinvestigate the use of observation systems. We will see that such systemscan increase transparency in the feedback process by making the focusof the observation explicit from the start.

Case for analysis: Wrong place on the audiotape

You observe an experienced language teacher working with six lower-intermediate learners in a 50-minute listening and speaking lesson.Before the lesson, the teacher tells you the lesson will begin with agrammar review, which should last 15 minutes. Then the students willhear a tape-recorded passage from the audio materials that accom-pany the textbook. They will identify the main idea and note the datesof key events. She tells you they will then talk in pairs, using the targetlanguage to compare their answers on a worksheet.

You take field notes during the lesson. After about five minutesof administrative chores, the grammar review begins. It lasts 20 min-utes and seems to be very teacher fronted. Although the teacher asksmany questions, the students don’t appear to be involved in the les-son. In order to document the students’ verbal participation in thelesson, you quickly sketch a seating chart. On it you keep track ofwhich students the teacher calls on, who asks questions and makescomments, and which students speak to one another.

When the listening exercise starts, there are some problems withthe way the tape recording has been set up. The teacher can’t seem tofind the segment of the audiotape that she wishes to play. The teacher

99

Language Teacher Supervision

asks the students to work on a written exercise while she tries to sortout the difficulty with the audiotape. Eventually the teacher does playthe listening passage, but the class period ends before the studentscan complete all the tasks.

The students thrust their belongings into their book bags and hurryout of the classroom, talking eagerly in their first language. No onesays good-bye to the teacher.

As the next group of students quietly enters the classroom, youthank the teacher and make an appointment with her to discuss thelesson. You then find a quiet place to flesh out your field notes andcheck the data you collected using the seating chart.

To avoid overwhelming the teacher in the post-observation con-ference, you decide to focus first on the seating chart data aboutthe interaction during the grammar lesson. You will then talk with herabout the problem with the audiotape, using the data from your fieldnotes as the starting point for the discussion.

Generating field notes as classroom data

In anthropology, field notes are written records of events observed in thefield. Depending on the phase of the investigation, these notes may beopen-ended or very tightly focused. These days, easy access to electronicrecording devices, including easily transported battery-operated units,means that we are no longer restricted to the laborious production ofhandwritten notes. But there are times when generating field notes canbe very useful in collecting data, whether those field notes are recordedwith paper and pen or on a laptop.

When might field notes be more useful than other data collectionoptions? Here are several possible situations:

1. When the teacher asks the supervisor to focus on one thing in parti-cular

2. When the supervisor wants to focus on something specific that is notcovered in an observation instrument

3. When video or audio recording equipment is not appropriate (e.g.,the noisy environment at a workplace language program in a factory)

4. When video or audio recording would be too intrusive or culturallyinappropriate

5. When there is no space in the classroom for the equipment6. When recording equipment is not available

It is also highly desirable to make field notes to accompany audio orvideo recordings, since both those approaches have limitations too.

100

Manual data collection procedures

Some key concepts in the use of field notes

One concern about a supervisor creating field notes is that the humanobserver is certainly more subjective than video or audio recordingswould be. For this reason, if you work with open-ended field notes,it is important to keep in mind the differences among observations,inferences, and opinions. Bailey et al. (2001:165) offer the followingdefinitions:

1. Observations: recording facts and events (e.g., in a scientificstudy); the data resulting from the observational act; com-ments or remarks based on something observed. (Please notethat here the term observations is used to encompass the indi-vidual products – the data – produced by the process of obser-vation.)

2. Inferences: decisions or conclusions based on somethingknown; ideas derived by reasoning; decisions based on factsor evidence; conclusions or deductions.

3. Opinions: beliefs not necessarily based on absolute certaintyor factual knowledge, but on what seems true, valid, or prob-able to one’s own mind; evaluations, or impressions of thevalue of a person, practice, idea, and so on.

These three types of records will all occur in your field notes, but it isimportant to distinguish among them, both in your mind and in yourdata. For instance, when I record an inference or an opinion in my fieldnotes, I flag it with OC, to indicate an observer’s comment.

Systematic field notes can provide a running commentary on the eventsduring lessons. But the field notes must be carefully prepared and suf-ficiently detailed to be clear and convincing. The supervisor must bedisciplined and responsible in recognizing the difference between data-based observations and inferences (or even opinions). This does notmean that the observer should avoid inferences or opinions entirely.Instead, the supervisor must (1) recognize inferences or opinions, (2)support them with verifiable observational data, and (3) check themagainst the teacher’s understanding of the same events. Field notes pro-vide “a human, interpretive dimension to observational data” (Bailey,2001a:118), which is often missing from videotapes, audiotapes, or struc-tured observation instruments.

One difficulty in generating good-quality field notes is that life in class-rooms moves very quickly. To capture detailed data, including directquotes and nonverbal behavior, note taking must be fast and accurate.Some people like to use laptop computers for generating field notes, butthe presence of someone typing in the classroom can be distracting toteacher and students alike (unless the students are also using laptops).

101


For this reason, I encourage supervisors to develop their own shorthandsystems for recording observational data.

Using a shorthand system for note taking

Some supervisors and researchers (see, e.g., Hunter, 1983) use system-atic abbreviations to save time while taking observational notes. Hereare some common symbols that are very useful in observing languagelessons:

@ = at ÷ = divide (divided by) % = percent# = number W/ = with → = goes to

& = and AM = morning > = greater than+ = add(ition) PM = afternoon or evening < = less than− = subtract(ion) e.g. = for example = femalex = multiply (multiplication) i.e. = that is = male

Other abbreviations are not so widely used but can be very helpful fornote taking in classrooms. Here are some abbreviations I’ve found usefulfor recording nonverbal behavior:

LH = left hand LA = left arm T = teacherRH = right hand RA = right arm St = studentBH = both hands BA = both arms Ss = students

th = thumb LL = left leg CB = chalkboardif = index finger RL = right leg Q = question

mf = middle finger LKn = left knee NV = nonverbalrf = ring finger RKn = right knee Rt = rightlf = little finger LF = left foot Lt = left

EC = eye contact RF = right foot T1, T2 = table 1, table 2

Still other shortened forms provide just enough of the word for the notetaker to recognize the symbol as unique. Here are some examples thatarise often in classroom observations:

wr = write lgh = laugh cn = candr = draw (p) = pause cd = coulder = erase (sm) = smile shl = shall

bk = book pprs = papers shd = shouldrd = read dsk = desk wll = willpt = point el = elicit wd = wouldy = you sil = silence m = mayd = do info = information mt = might

ans = answer diff = difficult mst = must

The symbols (p) and (sm), representing pause and smile, are in paren-theses because people often smile and typically pause in the middle of an

102


T. tells 6 Ss re: tape in TL. Ss lstn for mn idea & dates. All Ss have pprs—T-preprd cloze task handout. T strts tape rec. Realizes not at rt spot. Stps—rewinds. Ss wait sly. T starts tape agn. Not rt place. T ffwds tape. Starts.Not rt place. Triesagain 2X. T. tlls Ss wrk prs ex. in bk. (OC: T confused? flstrd?) Some Ss talk q'lyto others. 2 lks in bk bg. 1 asks ˝Why d w have to wrk in prs? T˝Oh. (p) ˝Oh, okay. Uhm, well w cn, hm, lets see. If y dnt want to b inprs, y cn wrk alone. T. tries the tape agn. Too sft. Turns up vol. 3 Ss lk up qkly (OC: Annoyed? Strtld.) as voice on tape ldly intrudes. 2 Ss pr up, 4 wrk s´ly alone.

+

Figure 5.1 Rough field notes

utterance. The parentheses distinguish these symbols from the ongoingquote in which they appear. The point is not that you learn my shorthandsystem but that you see how one works, so that you can try out a systemof your own.

Figure 5.1 gives an example of five minutes’ worth of rough notesfrom a classroom observation. Using the key above, try to interpret thesenotes.

If you take handwritten field notes, it is a good idea to schedule somequiet private time immediately after the observation to flesh them out.This can be done by entering them as a word-processed computer file,expanding on the original notes and abbreviations as you type the data.An advantage of creating computerized reports is that the database canquickly be searched for key words if you want to look for patterns inseveral reports or locate all the instances of a certain behavior or eventin a particular data set.

As an alternative, you can elaborate your rough notes by writingdirectly on the original, using a contrasting color of ink, so that youcan preserve the distinction between the original rough copy and thesubsequent elaboration. Then you can type or word process the elabo-rated notes as soon as possible, but preferably within 24 hours of theobservation. Figure 5.2 shows the elaborated version of the rough notesfrom Figure 5.1.

This small segment of field notes presents a snapshot of a lesson. Itis full of discussable material. In sharing these notes with the teacher, Iwould ask her first if the record seems accurate and read through it withher, pausing for her to comment on any points of particular interest. Iwould also ask her to interpret any observer’s comments for me, and Iwould ask about the pair work: Were the students accustomed to workingin pairs? What was the activity she wanted them to do in pairs? Howdid working in pairs relate to the tape-recorded material? The rest ofthe discussion would emerge largely from the answers to these sorts ofquestions.

103


The teacher tells the six students that they will be listening to a tape in the targetlanguage. They are to listen for the main idea and dates the speaker may say. All thestudents have papers. The teacher prepared a cloze task as a handout. The teacher startsthe tape recorder but she realizes that the tape is not cued up at the right spot. She stopsand rewinds the tape. Meanwhile, the students wait silently. The teacher starts the tapeagain, but it is still not cued up at the right place. The teacher fast forwards the tape andstarts to play it again. Once more it is not in the right place. She tries again twice more.The teacher then tells the students to work in pairs and do the exercise in the book.(OC: As the observer, I wonder if she is confused. She seems a bit flustered.) Somestudents talk quietly to others. 2 looks in her book bag. 1 asks ˝Why do we have towork in pairs? The teacher says, ˝Oh, and then pauses. ˝Oh, OK. Uhm, . . . well, we can, hm, let’s see. If you don’t want to be in pairs, you can work alone. The teachertries the tape again. The volume is too soft and she accidentally turns it up too loud.Three students look up quickly as the voice on the tape loudly intrudes. (OC: Are theyannoyed? Perhaps they are just startled.) Two students pair up, while the four otherswork silently alone.

+

Figure 5.2 Elaborated field notes based on rough field notes in Figure 5.1

Selective verbatim recording

In a technique called selective verbatim recording, the supervisor writesdown exactly what the lesson participants say (Acheson and Gall,1997:73), rather than paraphrasing or summarizing, so no informationis lost. For example, the paraphrase “T. told S. to sit down” could rep-resent statements from “Can you please sit down now so we can getstarted?” to “You sit down now! Don’t get up from your desk again!”The two statements mean the same (the teacher wants the student to sitdown), but the affective message differs.

This technique is selective in two ways. First, it is almost impossible torecord manually all that everyone says in a classroom, perhaps especiallyin interactive, student-centered classrooms. Second, the teacher and thesupervisor choose before the lesson what types of verbal events will becaptured while the lesson is in progress.

The selective verbatim technique is based on the assumption that the“learning process is heavily influenced by how students and teachers talkto each other. Therefore, teachers can learn how to improve their instruc-tion by careful analysis of their communication patterns” (ibid.:73).In our profession, the supervisor who wishes to use the selectiveverbatim technique in observing a language lesson will also need to beproficient in the target language.

There are four advantages of the selective verbatim technique, accord-ing to Acheson and Gall. First, it focuses teachers’ attention on whatthey say to their students (and what those students say to them). Second,because the data are selective, teachers can focus on specific verbal behav-iors without feeling overwhelmed. Third, the protocol produced in a

104


selective verbatim recording consists of direct quotes, so the data areobjective and nonjudgmental. Fourth, this simple, “low-tech” data col-lection procedure requires only a pen and paper.

This procedure is intentionally selective, so whatever is not beingfocused on can get lost. The data must be interpreted relative to theglobal context of the lesson (ibid.). And, knowing in advance what willbe recorded may cause the teacher to use (or avoid using) certain behav-iors. Thus, the data collection procedure can become an instructionalintervention (which is not necessarily bad in developmental observa-tions). Another concern is that the focus chosen for selective verbatimdata collection could be something trivial, which is easy to record but willnot influence learning. Therefore teachers and supervisors using the selec-tive verbatim technique should choose the observational focus carefully.

Data collection using pictorials and sketches

Many times teachers’ and learners’ nonverbal behavior is important in alearning event. But recording nonverbal behavior without a video camerais especially challenging.

We do have convenient labels for some facial expressions (e.g., scowl,frown, grimace, grin, squint, smirk, and smile) and some other nonver-bal behaviors. For instance, think about the following behaviors: “Theteacher stood facing the class with his arms at his sides, bent at the elbow,palms up. He arched his eyebrows while his lower lip protruded slightly,as he briefly raised and lowered his shoulders.” This entire descriptioncan be summed up in one word: shrug.

However, there are many nonverbal behaviors that are not so easilylabeled. For example, I once observed a lesson in which the teacher wastrying to get a student to produce the phonemic distinction between thesoft L and the hard L in Russian. She listened to the student’s pronuncia-tion and shook her head. She repeated the model, but again he producedthe form incorrectly. Eventually she placed her left hand above her right,palms turned downward, with her hands touching at the wrists. Holdingher left hand slightly cupped but still, she raised and lowered her righthand, while she pronounced the two phonemes in alternation. Mean-while, I was trying to record this fascinating gestural teaching device,having decided that the teacher’s left hand represented the roof of themouth and the right hand the relative tongue positions used in producingthese two sounds. I drew a quick sketch of the teacher’s hands, becausedrawing was a faster and clearer way to collect data than note takingwould have been.

I use this anecdote to show that sometimes drawing a picture is quickerand more descriptive than hurried writing might be. For example, the sim-ple image below describes a teacher’s orientation toward the chalkboard

105


T

Figure 5.3 Sketch of a teacher’s body position and eye contact

and the students. This quick sketch says, “The teacher stood perpendicu-lar to the chalkboard, left shoulder to the board, writing on it and lookingat it, but occasionally looking back at the students.” The horizontal linerepresents the board, and the vertical line the teacher’s body. The solidarrow indicates where the teacher’s gaze was primarily directed, and thedotted arrow where it was occasionally directed. In my notes I makethese simple drawings, along with a time notation, to indicate how oftenteachers change their stance and establish or break eye contact with theclass.

Why would it matter how a teacher stands relative to the chalkboardor other communal writing surface? Sometimes students become inatten-tive when the teacher breaks eye contact with the group. Of course, eyecontact differs from one culture to another, and some students remainattentive when the teacher turns away. Yet in my experience, data-basedfeedback of this sort has often been helpful to teachers concerned abouttheir classroom management.

Data collection using seating charts

There is a tradition in general education research of using classroom seat-ing charts as data collection mechanisms. Acheson and Gall (1997) illus-trate several uses of what they call Seating Chart Observation Records,or SCORE techniques.

The advantages of SCORE data are that teachers are very familiar withseating charts and can quickly interpret the records. SCORE data alsoprovide information about individual students, small groups, or the entireclass. Acheson and Gall discuss using SCORE procedures for studyingstudents’ time on task, verbal flow, and movement patterns within theclassroom. Figure 5.4 shows a simplified example based on the datafor the teacher and the six students in the case at the beginning of thechapter.

The data in Figure 5.4 say that the teacher asked 30 questions, threeof which were directed to individual students (I). The remaining 27questions were general solicits – questions or tasks posed to the group,rather than to individuals. The student data reveal that female student

106


I I I

F1 F2 F3

M1

I I

F4

O

M2

(Teacher’s zone)

(Students’desks)

Figure 5.4 Sample SCORE data for a 20-minute grammar review lesson

1 (F1) responded to six of the general solicits, and male student 2 (M2)responded to two general solicits. M1 responded to one general solicitand two individual solicits (also called personal solicits or direct nomina-tions) that were directed to him. F4, on the other hand, did not respondto any of the general solicits, and she also did not respond to a personalsolicit directed to her. The two remaining female students in the frontrow spoke more to one another than to anyone else. F2 answered twogeneral solicits but also spoke to F3 four times. F3 spoke to F2 five timesand responded to one general solicit. The key to interpreting these datais shown in Figure 5.5.

The SCORE data depicted in Figure 5.4 give information about whoinitiates verbal turns and who responds to them. This simple systemprovides fast records of who speaks to whom and how often. Of course,these data tell us nothing about the length or content of the turns. Nor dothese data convey any information about the participants’ target languageuse, their accuracy, or their fluency. A SCORE chart simply provides afrequency count of turns taken. Nevertheless, for some purposes andcontexts, SCORE data are quite helpful and informative.

Once an observer is familiar with the SCORE technique, seating chartscan be used to record simple symbols depicting the interaction of largergroups as well. In addition, SCORE data can be collected on just one ortwo individual students. The narrow focus can yield very helpful infor-mation if a teacher is concerned about particular students.

Data collection using classroom maps

Simple maps can also be used to collect classroom data. Maps can helpyou identify each student, detect interaction patterns, define territories

107


M or F = Male or female I = Individual student responds to apersonal solicit from the teacher

= Teacher’s general solicit tothe class

O = No response from the student to apersonal solicit from the teacher

I = Teacher’s personal solicit toan individual student

= Student response to a singlegeneral solicit from the teacher

or = Student speaks to anotherstudent

= Student has responded to fivegeneral solicits

SYMBOL KEY SYMBOL KEY

Figure 5.5 Key to SCORE data symbols

within the room, and locate possible distractions (e.g., traffic noise out-side a window). I sketch a map using the following techniques wheneverI observe a language lesson, adding the names of the students if theyintroduce themselves or when I overhear someone call them by name.This strategy allows me to talk more specifically with the teacher afterthe lesson and ask questions about individual students.

Figure 5.6 presents an actual map from an observation of a mathteaching assistant in a university setting. I use these data here, eventhough the teacher’s subject matter is math rather than language, becausethe particular map reveals a great deal about the environment of theroom, and it suggests some simple ways in which the teaching could beimproved.

In this map, TR stands for tape recorder; Obs, observer; OHP, over-head projector; �, empty desk; , male student; and , female student.The map is part of a large data set in which I observed 24 teachers work-ing with more than 400 students, so I wasn’t able to learn all the students’names. Still, I needed some way to identify each student within each les-son I observed. The strategy I adopted was to identify the student nearestthe door at the front of the room as the first person in row one (e.g.,

1,1). The student next to her in the same row was the second personin row one, and so on.

Before you read further, please answer the following questions: Whatdo you know about this lesson, based on the data in the map alone?What questions does the map raise in your mind?

The teaching assistant (TA) for this class was a native speaker of Chi-nese, whom I will call Kwan (a pseudonym). The class was a “discussionsection” – a regularly scheduled meeting following a professor’s lecture,during which students can ask questions and discuss the material with

108


1,1 1,2

2,1 2,2 2,3 2,4

3,1

T’s

Unused Chalkboard

ObsTR

OHP

Chalkboard

Chair

Door

+

+

Windows

Figure 5.6 Map of a math class taught by a teaching assistant

the TA. There were seven students present, and 35 empty desks. The legsof the desks were clamped together, so that where two or more desks arepictured as adjacent, they are actually aligned in cramped, fixed rows. Iwas observing the TA for research purposes, rather than as his supervi-sor, so I am reprinting these data here only to illustrate the use of mapsand field notes. We will return to the information contained in this mapafter learning more about Kwan.

Creating profiles of teachers or learners

One helpful use of field notes is to create a profile of a teacher. To generatea profile or “typical lesson” report, the full-length field notes can besummarized. Three or more sets of summarized notes on one teachercan be combined to generate a profile of the teacher, by looking for thenotable similarities across the various summaries.

This profile of Kwan (Bailey, 1984:114–115) is based on three hours ofobservation over a 10-week term. (This class met for a total of 20 hoursthat term, so three hours represents 15 percent of the in-class time.) Eachset of field notes was condensed into a two-page synopsis. The threesummaries were then coalesced to create the following profile:

Kwan was a native speaker of Chinese who was the TA for amath discussion section. There were seldom many students inhis section, and those who did come tended to trickle in late and

109


leave early. At the first meeting there were 18 students present.During two other observations there were only three studentsone day and seven on another – which happened to be the daybefore an exam. At one of my scheduled observations, no stu-dents appeared at all. These low attendance patterns may berelated to Kwan’s teaching style.

At the first class meeting, Kwan wrote his name, office hours,office number, and the course number on the board, but he toldthe students nothing about his background and asked nothingabout them (e.g., he apparently did not try to learn their names).There were no “getting-to-know-you” remarks – no commentsabout the course or himself to put the students at ease. In short,Kwan made no effort to build rapport with the students.

Kwan’s classroom style was rather passive. Much of thisimpression was based on nonlinguistic signals. He seldom usedgestures to support the meaning of his speech. When he was notwriting, his arms hung limply by his sides – and even when hewas writing, his other hand was inactive. His voice was soft, andhe sometimes mumbled or talked toward the blackboard. Therewas little or no difference between the volume or pitch that heused as he spoke to the students and that he used as he did cal-culations on the blackboard. He did not project his voice whenaddressing the class; it was as if he were talking to himself. Therewere also periods of silence as the TA wrote on the board, readhis math book, or awaited students’ questions.

Kwan didn’t call on students during any of the classroomobservations. Instead, he would stand silently, waiting for themto ask questions. Then he would usually work problems (ratherthan explaining them) in response to their requests. Writingseemed to be his main channel for communicating with the stu-dents. Indeed, he had what could be called “good blackboardtechnique” (i.e., clear handwriting, organized layout of solu-tions on the board, boxes drawn around the answers, verticallines drawn to separate completed portions of problems fromnew work, etc.). But as a result of Kwan’s involvement with theblackboard, there was relatively little eye contact between himand the students. Instead he would look at the problem and occa-sionally glance back at the students, with the blackboard holdinghis attention.

Now that you’ve read Kwan’s profile, please glance back at the map ofhis classroom in Figure 5.6. Each time I observed Kwan, I drew a mapof the room. The three different maps revealed two important bits ofinformation. The first had to do with the number of students present,

110


relative to the number of chairs available. There were eighteen, three,and seven students present in Kwan’s class on the three days I observed,so I wondered why his section had been scheduled in such a large room.This issue might not have jumped out at me if I had simply recordedthe number of students present at each lesson without drawing a map.However, the number of empty seats was a visual fact that could not beignored. I became curious about how many students were supposed tobe attending Kwan’s discussion section. The Math Department secretarytold me that there were 35 students enrolled.

Another interesting datum was the overhead projector in the corner.The maps show that it had indeed been present at all three observations,but Kwan never used it when I observed his class. If I had been Kwan’ssupervisor, we could have discussed the option of using the overhead pro-jector. If the math problems had been written on transparencies beforethe class, he might have been able to talk through the steps of the solu-tions, thereby avoiding the long periods of silence in his “discussion”section.

Using field notes, a profile can reveal stable behaviors over time orto generate a typology of teachers or learners. This process is depictedin Figure 5.7. I used this process in the research on the communication

Profile of Teacher 1







Type A

Type B

Type C

Observation 1 of Teacher 1



Summary 1 Summary 2 Summary 3

Figure 5.7 Summaries, profiles, and teacher types

111


problems of non-native-speaking TAs in U.S. universities (Bailey, 1984).Five clear types emerged from the data: (1) the inspiring cheerleaders;(2) the entertaining allies; (3) the knowledgeable helpers and casualfriends; (4) the mechanical problem solvers; and (5) the active unin-telligible TAs. (Kwan’s profile illustrates type 4, the mechanical prob-lem solvers.) These five labels were not predetermined categories takeninto the observations. Instead, they emerged as the TAs’ profiles werecompared.

Using observation instruments

One means of gathering data in classrooms is to use a paper-and-pencilobservation instrument (sometimes called an “observation schedule”).Gebhard notes that the use of such observation systems “allows super-visors to describe rather than prescribe teaching” (1984:509).

An observation instrument is essentially a paper form containing pre-determined categories that guide the observer’s collection of informationor the judging process. The categories can be presented as questions toanswer or topics to address in writing. Another common format con-sists of statements with which the observer rates performance (e.g., on ascale of 1 equals “poor,” 5 equals “excellent”), or statements followedby evaluative categories (such as “superior,” “satisfactory,” or “needsimprovement”) for observers to circle or check.

Observation instrument categories range from “low-inference” to“high-inference” statements (Long, 1980). A high-inference statementrequires judgment while the observer is recording (e.g., rating theteacher’s organizational skills). Any high-inference category system likethis yields data, but note that the data are not a record of the lesson events.Instead, they are a record of the observer’s opinions of those events. Incontrast, low-inference statements are verifiable (i.e., they are observa-tions rather than inferences or opinions), so less judgment is called foron the part of an observer. For example, the statement “positive socialclimate throughout lesson” is closer to the high-inference end of thecontinuum than is a category that has the observer tally every time anindividual student smiles during a five-minute period.

Observation instruments embody theories of language learning andteaching. The statements or categories in the instruments represent thedesigners’ beliefs about what factors are important. Often these theoriesare not made explicit. So one important question to ask in selecting anobservation instrument is what theory of language learning or teachingthe instrument is based on. Knowing that, supervisors and teachers candecide whether the instrument is appropriate. An observation instrumentbased on the behaviorist principles of the audiolingual method would

112


entail coding or rating behaviors (immediate public treatment of oralerrors, for instance), which would not be appropriate in an observationinstrument predicated on communicative language teaching and interac-tionist theories of language learning.

Adapting or designing observation instruments

If an observation system is to be used, one can be chosen from amongthe available formats, or the supervisor and the teachers could borrowelements of existing systems and adapt them to fit local needs. Such adap-tations could include adding or deleting categories, or revising descrip-tors to make them more appropriate to the context. The team could alsoadd a numerical rating system or spaces for open-ended comments if theexisting system lacks these options.

Teachers can also devise their own observation system. This optionis very time-consuming, but the process generates important discussionsabout teaching values. Creating an observation system may also promote“buy in” among the teachers. Murdoch (2000) notes that to empowerand motivate teachers, supervisors should involve them in developingevaluative data collection processes. Designing an observation instru-ment is one way to do so.

If the members of a teaching staff design an observation instrument,it can be based on the goals, philosophy, and content of the specificlanguage program. If, for example, a goal of a particular course is todevelop students’ fluency and confidence in speaking the target language,it would probably be inappropriate to use categories that rate teacherson their immediate treatment of oral errors. Obviously, designing suchan instrument is more straightforward if the program goals are alreadystated. If they are not, then before teachers can be evaluated on how wellthey are implementing the program goals, those goals must be articulatedand agreed on. In addition, it is important to pilot the instrument innonbinding practice observations before collecting data that will be usedin any official teacher evaluations.

Using existing observation instruments

There is a long tradition of using observation systems to record infor-mation about teaching and learning, and many systems are available.In fact, more than two decades ago, Long (1980) contrasted 22 suchsystems on nine dimensions: type of recording procedure; high, low,or mixed inferences; number of categories used; whether multiple cod-ing is permitted; whether real-time coding is involved; source of thevariables; intended purpose (research and / or teacher training); whetherthe unit of analysis is time or an analytic unit of some kind; and the

113


focus – the “range of behaviors and events sampled” (p. 4). Other sys-tems have been developed since Long published his analysis, but his cat-egories for comparison are still useful. Several observation systems arereprinted in Allwright and Bailey (1991), as well as in Day (1990), soindividual formats will not be discussed in detail here.

Many observation instruments were originally designed in the 1960sand 1970s for research purposes, rather than for language teachersupervision. Some, like the Embryonic Category system (Long, Adams,McLean, and Castanos, 1976), were intended for coding data transcribedfrom audio- or videotapes, rather than for “real-time” coding (live cod-ing during a lesson). Some instruments for observing language lessonsare retoolings of systems originally designed for general education class-rooms. For example, the well-known Interaction Analysis system (Flan-ders, 1970, 1976) from general education led to the Foreign LanguageInteraction system or “FLInt” (Moskowitz, 1966, 1968, 1971).

In a study with student teachers utilizing FLInt, Moskowitz stated thatthe “main goal in training the teachers in observational systems was toincrease their sensitivity to their own classroom behavior and its effectsand influence on students” (1968:231). In other words, awareness-raisingwas a key goal. In reviewing 18 projects in which Interaction Analysiswas used to help modify teaching behavior, Flanders wrote, “If thereis any common bond which cuts across all of the projects . . . it is thatthey all provide some support for the following proposition. Attentionto teaching behavior, practice in analyzing it, and performing it withfeedback tends to incorporate such behavior in the teacher’s repertoire”(1970:352; italics in the original). The assumption underlying the useof such observation systems is that feedback is important in changingteaching behavior. Jarvis concluded that “the very use of such instrumentsmay, in fact, be a causal factor in behavior change” (1968:341).

Some instruments have a specific theoretical orientation. One exam-ple in our field is COLT: the Communicative Orientation of LanguageTeaching (Allen, Frohlich, and Spada, 1984; Frolich, Spada, and Allen,1985). COLT was designed “to capture differences in the communicativeorientation of L2 classroom interaction” (Frolich et al., 1985:27). Thetwo parts of the instrument reflect this orientation. Part A “contains cat-egories derived primarily from pedagogical issues in the communicativelanguage teaching literature” (ibid.:29), including activity, group work,content, topic control, student modality, materials, and the like. Part Bof the instrument documents the verbal interactions that occur duringclass activities. The categories in Part B are derived from first and secondlanguage acquisition research findings.

In short, there are many observation instruments available for researchor teacher training purposes, though several are quite old and may not

114


T O T O T O

Figure 5.8 Levels of congruity in the teacher’s and the observer’s focus in anobservation

reflect current pedagogy. Language teacher supervisors may use or adaptthese instruments to guide their data collection. However, many questionsarise in selecting such instruments: Is the focus consistent with the indi-vidual teacher’s goals for improvement and the supervisor’s reasons forobserving the class? Are the assumptions underlying the instrument clearto the teachers and supervisors? Are the instrument’s theoretical under-pinnings appropriate as evaluative guidelines for this program’s curricu-lum and teaching policies? Does the observation instrument provide amechanism for collecting appropriate data in this particular program?Will the instrument yield useful data for answering questions about lan-guage teaching and learning? A supervisor should consider all these ques-tions in selecting or adapting an existing observation system.

Using observation instruments to specifythe observational focus

Using an agreed-on observation instrument can serve an important pur-pose in pre- and post-observation conferences. Observers bring concep-tual “lenses” into the classroom, regardless of whether they use obser-vation systems. An observer generating field notes, for example, mayfocus on issues of interest to him or her, based on his or her beliefsand background knowledge. Using an observation instrument may helpexternalize and systemize that knowledge and those beliefs. Communi-cation between the teacher and supervisor can be enhanced if the teacheris familiar with the observation instrument used and agrees that it isappropriate. It is even better if the teacher is skilled in its use and thusunderstands what the supervisor is doing during an observation.

Making both participants’ perspectives explicit before the observationcan help to focus the feedback on topics of interest to both the teacher(T) and the observer (O). Three possible levels of congruity are depictedin Figure 5.8.

The first pair of circles is reminiscent of both Gebhard’s (1984) andNunan’s (in Bailey et al., 2001) personal accounts of supervisors focusing

115


on something quite different from the focus each of these teachers hadchosen. In the second pair of circles, the shaded area indicates partial con-gruity between the teacher’s interest and the observer’s, while the thirddiagram indicates a high degree of overlap. The use of an agreed-on obser-vation system, or clear focus setting, in the pre-observation conferencecan promote such congruity.

In my opinion, teachers should have a major role in devising or choos-ing any instrument to be used and in directing the supervisor’s focus dur-ing observations. Consensus about the instrument and familiarity withits categories can help teachers incorporate teaching elements studiedin teacher preparation courses or in-service training. Using an acceptedobservation system seems to lower anxiety (because there are fewer sur-prises), which in turn allows the teacher to consider the observer’s ideas.

Advantages and disadvantages of usingobservation instruments

Williams explains why many people dislike the use of observational cat-egories. First, for the teachers in Williams’s context, observation witha checklist “was threatening, frightening, and regarded as an ordeal”(1989:86). Second, the teacher had no responsibility for the evaluationprocess. It was trainer-centered, even though the program emphasizedchild-centered teaching (ibid.). The checklist was prescriptive, and teach-ers felt it focused on too many things at once (ibid.). There also seemedto be no continuity across three classroom visits. Finally, the establishedcategories didn’t accommodate the individual teacher’s pace or wishes(ibid.).

The problematic use of decontextualized checklists was summarizedby Chamberlin, who wrote that a supervisor typically “observes theteacher’s behavior and rates behaviors on a standard scale of measure,which is then placed into the teacher’s file. This approach mirrors anattempt to reduce teaching to a technical act that can be measured bya set of prescribed criteria” (2000:654). Chamberlin’s concern abouta “set of prescribed criteria” can be partially overcome if the teach-ers themselves either revise or create the categories, but the point thatobservational categories are often used out of context and can reduceteaching to a “technical act” is still a matter of concern. Furthermore,“almost without exception, such technicist approaches to teacher evalu-ation fail . . . because teachers and students do not live their lives in thefragmented and dislocated ways suggested in the observation schedules”(Gitlin and Smyth, 1990:85).

There are both advantages and disadvantages to using observationinstruments in language teacher supervision. The advantages are all based

116


on the fact that the predetermined categories don’t change from oneobservation to another. Teachers and supervisors know from the outseton which issues to focus because those features are codified in the par-ticular instrument’s categories. Second, ratings on nominal categories ornumerical scales can be compared over time, with improvement plottedin subsequent observations. Third, teachers and supervisors can focus onvarious categories in any given lesson and disregard or downplay the oth-ers. For instance, if an instrument highlights promoting student-studentinteraction, the teacher can plan opportunities for such interaction andattend to this feature during the observed lesson. Finally, observationinstruments embody an established, finite number of variables, so teach-ers may feel they are not facing an unending task or aiming at a movingtarget.

Ironically, the disadvantages of using observation instruments in lan-guage teacher supervision are based on the same facts: The categories onthe instruments are predetermined and unchanging. Of course, some ofthe observation systems (e.g., COLT) include a space for recording open-ended comments, which introduces some flexibility in the data collec-tion process. Nevertheless, supervisors should be aware of the followingpotentially problematic issues. First, the theoretical position(s) embodiedin the instrument may not match the position(s) held by the teachers orespoused by the program. Second, the topics covered in the instrument’scategories may not match the issues that arise during a particular les-son. And third, the categories may not match the teacher’s professionaldevelopment issues at the time. In addition, some people argue that ifteachers know what criteria (in the form of an observation instrument)will be used, they may teach to the category system instead of teachingas they usually do. This could be either an advantage or a disadvantage,depending on the purpose of the observation.

We saw in Chapter 2 that the teacher’s focus and the supervisor’s focusin an observation can differ widely. This problem can be exacerbatedif the teacher is unfamiliar with or doesn’t value the observation instru-ment used by the supervisor. If the instrument makes the supervisor focuson the teacher’s grammar presentation, but the teacher is concentratingon managing group work, then the feedback from the supervisor mayfail on two counts. First, it doesn’t address the area in which the teacherwanted input (i.e., the desired input is not received). Second, the feed-back that is provided may be discounted as irrelevant or unhelpful if itaddresses a topic the teacher feels is unimportant at the time (i.e., theinput that is received is not desired).

Given these shortcomings, it is important that a supervisor or teach-ing team carefully analyze any existing instrument to determine whetherits underlying philosophy is consistent and appropriate in the program

117


context. Using an observational system that is not appropriate can createserious problems in teacher evaluation and staff morale.

Concluding comments

This chapter examined two manual data collection approaches. Thefirst is the more open-ended, unstructured but systematic use of fieldnotes, maps, sketches, and seating charts. The second involves struc-tured observation instruments, whose categories range from objective tohighly inferential.

The following Case Discussion is based on the case that began thechapter. The excerpt from the field notes (Figure 5.2) describes a segmentof the lesson that you will discuss with the teacher. The SCORE data inFigure 5.4 are from the grammar review. You have given the teacher yourelaborated field notes and the SCORE data prior to the conference.

Case discussion

1. Please reread the field notes in Figure 5.2 and the SCORE data inFigure 5.4. Then decide what areas you can discuss with this teacher asstrengths. What issues will you discuss as areas for improvement? Areyou dealing with issues of awareness, attitude, skills, and /or know-ledge (Freeman, 1989a)?

2. How would you describe this teacher’s zone of proximal development?What are your short-term goals for working with this teacher? Whatare your long-term goals for this teacher in this program?

3. Think about the Johari Window (Luft and Ingram, 1969), reprintedas Figures 2.3 and 2.4 in Chapter 2. What kind of development doyou hope this teacher will experience as a result of your feedback?Give some specific examples from the case, using the Johari Window.

4. How would your conference with the teacher be different with andwithout the field notes?

5. Do the SCORE data (Figure 5.4) support your impression that thegrammar review was teacher-centered? What patterns do the datareveal that you could discuss with the teacher?

6. Role-play the conference with this teacher. What are the discourse dif-ferences if you use (A) the supervisory or traditional directive option,(B) the alternative option, and (C) the nondirective option (Freeman,1982; Gebhard, 1984) carefully and consistently in role-playing threepost-observation conferences? Are there some issues that could bediscussed better using one of these three options?

7. On what areas would you want to focus in a future observation ofthis teacher? Why?

118


Tasks and discussion

1. Suppose you are a language teacher whose supervisor observes yourclass. Afterwards, the supervisor gives you this grid to review beforethe post-observation conference:

Sample rating grid 1

Categories Poor Fair Average Good Excellent N. A.Category T XCategory U XCategory V XCategory W XCategory X XCategory Y XCategory Z X

What questions would you have? What would you want to tell thesupervisor?

2. Do your answers depend on the categories? For example, if you werethe teacher, would your reactions change if this rating grid includedthe following categories?


Categories Poor Fair Average Good Excellent N. A.Use of authenticlanguage samples XVariety inlearning activities XGroup work, pairwork, and studentinteraction

X

Development ofthe physicalenvironment

X

Appropriatefeedback to 2LLs XPositive affect inlesson XClarity ofinstructions andexplanations

X

119


3. You are the supervisor preparing for the post-observation conferencewith the teacher whose lesson you rated in Sample Grid 2. What wouldyou want to ask this teacher? What would you want to say to himor her? What evidence would you use to explain the ratings of fair,average, good, and excellent?

4. Now consider Sample Grid 3, the ratings made after three visits toone teacher’s class. The numbers 1, 2, and 3 represent the supervi-sor’s opinion at the first, second, or third observation. What patternsemerge in these data?


Categories Poor Fair Average Good Excellent N. A.Use of authenticlanguage samples

1 2,3

Variety inlearning activities

1,2

3

Group work, pairwork, and studentinteraction

1,2

3

Development ofthe physicalenvironment

1,2, 3

Appropriatefeedback to 2LLs

1,2

3

Positive affect inlesson

1, 2, 3

Clarity ofinstructions andexplanations

1 2 3

5. If you were the teacher who had taught these lessons, what patternswould you find most interesting? Most troubling? What comments orquestions would you have?

6. If you had compiled this grid, what patterns would you find mostinteresting? What particular information in the grid would you wantto discuss with the teacher? With a colleague, role-play the third post-observation conference and discuss these data.

7. In the profile of Kwan, identify all instances of observations, infer-ences, and opinions.

8. What does each of the following images convey to you? The horizon-tal line represents the chalkboard, while the shorter horizontal linerepresents the teacher’s orientation. The arrows show the direction

120


of the teacher’s gaze. A dotted arrow represents an occasional glanceand a solid arrow shows sustained eye contact.

A. B. C.

D. E. F.

Suggestions for further reading

Allwright (1988) published a history of several observation instruments.The book includes the original authors’ ideas as well as Allwright’s com-mentary.

Acheson and Gall (1997) devote four chapters to data collection pro-cedures, each of which is illustrated with clear examples.

Bailey et al. (2001) provide examples of observational field notes andtwo colleagues’ discussion about the information one gathered whilewatching the other teach.

Abbott and Carter (1985) include a sample checklist for observingtrainees’ practice teaching lessons. Omaggio (1982) provides an obser-vation instrument and sample data. Rhodes and Heining-Boynton (1993)give an observation form for language lessons in primary schools.

For discussions about the instrument called FOCUS, see Gebhard(1999), Gebhard and Oprandy (1999), and Oprandy (1999). FOCUSstands for “Foci for Observing Communications Under Settings,” aninfluential system designed by Fanselow (1977, 1987, 1988).

Day (1990) discusses ethnography, audio and video recording, seatingcharts, and other coding techniques. The use of seating charts for datacollection in language classes is illustrated in Gebhard, Hashimoto, Joe,and Lee (1999). See also Swan (1993).

121

Manual Data Collection

Documents

Transcript of Manual Data Collection