Project Design and Data Collection Methods:

A quick and dirty introduction to classroom research

Margaret Waterman

September 21, 2005

SoTL Fellows 2005-2006

Where are we inthe Action Research Cycle?

Identify Innovation

Gather Data

Analyze DataInterpret Data

Develop Action Plan

☻

Decide when and how to collect data

Design vs. Method Design is about when and from whom you are

going to collect data so that the data can be interpreted.

On Tuesday

With Colonel Mustard

Methods are about where and how you are going to collect that data.

In the Library

With a candlestick

With an interview

Designs: Experimental Experimental: strongest for drawing

inferences about cause and effect. Control vs. experimental groups, randomly

assigned, one variable at a time Diagram as follows:

O1 X1 O2

O1 O2

O = observation, X = treatment; Each row is a group

Experimental designs

Problems: Not always possible in education to have

two or more matched groups,

Random assignment usually not feasible,

May not be suitable for small-scale projects

Usually not for classroom research problems

Designs: Pre-Experimental Weakest designs: Post test only

X1 O1

Slightly stronger: Post-test with comparison group

X1 O1

O1

NOT RECOMMENDED

Pre-Experimental: Post Test OnlyProblems:

Too many other reasons for observed conditions Cannot draw valid inferences about causality

because you don’t know what pre-treatment condition is.

In comparison group: may have differences in incoming ability in the two groups

Good for: preliminary observation and development of hypotheses, not showing cause and effect

Designs: Pre-Experimental One group, Pre-test Post-test

O1 X1 O2

Uses: Can make some inferences about cause and effect, but only when discuss possible intervening variables.

Useful in classrooms when trying new techniques or materials.

Pre-Experimental: Pre-test, Post-test Problems:

Can’t discount all intervening variables: e.g., history, awareness of being in an experiment, selection effects on group membership.

Generalizability is limited May sensitize students with the pre-test

Designs: Quasi-Experimental

Uses an experimental design, but groups are not randomly assigned.

O1 X1 O2

O1 O2 Effect of treatment isolated

X1 O2 Use two more groups to

O2 isolate effect of pre-test.

Can make inferences with more confidence than with one-group designs

Problem: intervening variables again.

Design: Time Series

One group, multiple observations before and after treatment. Can be two groups or two treatments.

O1 O2 O3 O4 X1 O5 O6 O7 O8

OR

O1 O2 O3 O4 X1 O5 O6 O7 X2 O8 O9 O10

Look for BIG differences after treatment as compared to before treatment.

Problem: The intervening variables

Linking Design and Methods

O1 X1 O2

Suppose the design is one-group, pre-test, post-test:

How, where, and exactly when will the data be gathered at the two points of observation?

Action Research Often Uses These Kinds of Methods to Gather Data

Surveys

Observations

Interviews

Artifact Analysis

Methods of Gathering Data Make observations

Structured observations

Unstructured observations

Ask questions

Surveys

Interviews

Examine and Score Artifacts

Tests, portfolios, treatment plans, student comments

Must design scoring guide

Structured Observations Examples:

Use a class map to keep track of who asks questions, answers questions, makes comments.

Have a list of behaviors. Make simple hash marks for each type of behavior when it occurs.

Make a list, from a recorded class discussion, of the kinds of questions you asked.

Methods: Unstructured Observations Examples

A journal that a teacher keeps to record personal impressions of how a class is going.

A written remembrance of interactions with one or a few students that are being tracked over time.

A written set of impressions made while watching a videotape of a class.

Methods: Unstructured Observations Think of these as similar to ethnographic “field

notes,” a place to record reactions, impressions, planned actions, responses and thoughts.

Can be useful to see trends in how YOU are responding to the innovation you are attempting.

Can be useful when studying one or a few students, again, over time to see trends.

Methods of Gathering Data Make observations

Structured observations

Unstructured observations

Ask questions

Surveys

Interviews

Examine and Score Artifacts

Tests, portfolios, treatment plans, student comments

Must design scoring guide

Methods: Questioning (survey/interview)

Uses a prepared set of questions. May have open-ended or close-ended

questions, or a combination. Good for getting background info. Can be used pre and post treatment. Can address things that may not be visible,

such as attitudes, motivation. Use with individuals or groups.

Methods: Questioning with Surveys Examples:

A survey of attitudes toward science

Rankings of importance

Student ratings of instruction

May include open-ended questions, e.g., what element of this course most helped you to learn?

May include some content, but if entirely content, it’s a test and is an artifact of the course.

Sample from an Survey

Methods: Questioning with Interviews Advantage over surveys: can ask follow-ups,

more personal contact. Examples:

Sort readings by usefulness, telling why Solve a genetics problem aloud, explaining

thinking A focus group Interview of team members in small groups

Sample from Interview

Methods of Gathering Data Make observations

Structured observations

Unstructured observations

Ask questions

Surveys

Interviews

Examine and Score Artifacts

Tests, portfolios, treatment plans, student comments

Must design scoring guide

Method: Examine and Score Artifacts Examples of artifacts:

Diagrams of cells before and after instruction Wear on computer keys to see which are hit most Answers to a test question Portfolios Term papers Case analyses Management plans Treatment plans

Methods: Artifact Analysis

Decide what you kinds of materials you want to collect

Justify why the artifact you are choosing is a good choice given your research question

Create a scoring rubric (guide) to assign points

Good for pre-post designs

A KEY TO SUCCESS: Pilot Test Your Instruments

Give a small group of people (not in your class if you can) your survey or interview

Collect sample artifacts to see if your grading scheme works

Try out your observation scheme to see if it needs to be tweaked.

Designs and Methods: Summary Many of these approaches are familiar to we

who teach Much of this is used in research in many

disciplines. Talk about your design and instruments with

Teaching Associates, Fellows, other colleagues.

Keep your data collection focused. Try to keep the project reasonable for the

time you have available.

To create your own plans answer the questions below: From whom are you gathering data?

More than one class, subgroups?

When will you gather data?

All one term, over several courses, in three weeks?

How will you gather data?

Questions, artifacts, observations?

Where will you gather data?Classroom, online forum, mailed survey, telephone?