Designing SoTL Studies – Part IValidity

Class Session 5 – Henry Clark, PhD

Independent variable = the

cause

Dependent variable = the

effect

The researcher controls or

manipulates the independent variable (the treatment)

The dependent variable is what is measured, often

called the assessment

(knowledge, skills or attitudes).

Testing Hypotheses

A Simple Hypothesis : The treatment (independent variable) improves

students on the assessment (dependent variable).

Three possible major problems related to causality:

1. The assessment was not measured well

2. The treatment was not manipulated well

3. Something other than the treatment caused change in the assessment (internal validity).

Construct and Internal Validity

0Construct Validity:

Am I measuring what I think I am measuring?

Am I implementing what I think I am implementing?

0Internal Validity: Did the treatment cause the outcome?

A Simple Hypothesis : The treatment (independent variable) improves

students on the assessment (dependent variable).

Three possible major problems related to causality:

1. The assessment was not measured well (reliability and construct validity)

2. The treatment was not manipulated well (construct validity)

3. Something other than the treatment caused change in the assessment (internal validity).

A study does not have absolute validity or absolutely no validity

The level of validity relates to the confidence in the conclusions

Construct and internal validity are measured on a continuum

Construct validity does not imply internal validity (and vice versa)

When a hypothesis is supported, it does not necessarily mean that the

study has either construct or internal validity

Some notes on

evaluating construct

and internal validity

What is meant by “construct”?0A concept, model, or schematic idea

0A construct is the global notion of the measure, such as:

0 Student motivation

0 Intelligence

0 Student learning

0 Student anxiety

0The specific method of measuring a construct is called the operational definition

0For any construct, researchers can choose many possible operational definitions

To Improve Construct Validity of Measures

0 Measure learning directly (clear operational definitions; learning is not the same as enjoyment or perceived learning)

0 Measure student learning through student learning objectives (ensure these are aligned with assessments)

0 Use Established Scales to Measure Student Attitudes and Personality (Don’t reinvent the wheel; Tests in Print)

Good Measurement is Important To Improve Construct Validity

0Know How To Score the Measure (make sure you’ve established this before data collection; know what is reasonable; IOTT; rubrics; training; IRR)

0Determine Whether to Use Graded or Ungraded Measures (pros and cons of both)

0Minimize Participant and Researcher Expectancies

To Improve Construct Validity

0Determine Whether to Use Multiple Operational Definitions (can use multiple measures)

0 Use a Retention Measure to Investigate Long-term Effects (but treat long term results with caution about other influences)

Good Differences between Conditions Improve Construct Validity

The treatment (intervention) needs to be manipulated well to ensure

construct validity

The only difference between conditions should be the treatment

Other variables that are different between conditions are confounds

To determine construct validity, treatments need specific operational definitions

Anything that can affect the results and cause a difference between students in treatment and control conditions needs to be documented

Potential problems in using different sections of a class

Construct validity of the treatment is questionable in any design that

compares one section of a class with another

Classes are a social space, and the students and instructors are interdependent

Students can ask different questions

The class may have a different “tone”

Splitting a class into two groups can minimize this concern; if students in a split class can be randomly assigned to a condition, internal validity will increase

Different Types of Comparison in Research Design  Between

ParticipantsWithin Participants: Multiple Treatments

Within Participants: Multiple Measures

How comparison works

Students in one condition compared to students in another condition (control – Treatment; multiple T’s)

All students in both control and treatment conditions

Students receive both pre-test (control) and post-test (treatment)

Strengths No carryover effects from multiple treatments; no instrumentation or testing effects from multiple assessments

No selection bias; greater statistical power

No selection bias; greater statistical power

Weaknesses Selection bias without random assignment; many differences if groups are separate (e.g., two separate classes); lower statistical power

Instrumentation and testing effects; carryover effects

Instrumentation and testing effects; other confounds that occur between assessmens

Improve Internal Validity by:

Random assignment ; adding covariates

Counterbalancing Increase number of assessments; add no treatment separate control condition; use alternative measures for assessment

External Validity

Can the sample used in the study

generalize to other groups or

populations?

Generally, it is impossible in

classroom studies to get a sample

that will generalize to all

students.

The researcher should report demographic chacteristics

How realistic is the situation? In

a classroom, if the treatment works, external validity

is higher

Designing SoTL Studies – Part IIPracticality

Common Practical Problems in SoTL Research

Researchers who think they need to measure everything

Researchers who do not have many students: low statistical power

Researchers who only have a single class; limits to type of design

Difficulties in random assignment

Difficulties in determining whether the treatment is potent enough to have an effect (see power above)

Concerns about conducting an ethical study in a classroom or training situation

Don’t Use Want to make statement

about causality Have low number of

students

Use Have single group of

students that cannot be divided

Have only one session in which to collect data

Additional Options:Correlate many variables at the same time

Simple Correlation

One-Group, Post Test Only

Don’t Use Want to make statement

about causality

Want to make comparison to another group

Use Desired focus is on

describing treatment and not assessment

Cannot have pre-test or control group

Want single group of students that cannot be divided

Two-Group, Post-Test Only Don’t Use

Have low number of students

Groups are very different Have different assessments

for each condition

Use Concerned about carryover

effects Concerned about testing and

instrumentation effects Have multiple groups Have only one session to

collect data

Additional Options:• Use random assignment to improve internal validity• Add post-test to assess long-term change• Add additional conditions• Use covariates to improve internal validity and power

One Group, Pre-test, Post-test

Don’t Use Items other than treatment

occur between assessments First assessment affects

second Students likely to change

between assessments with no treatment

Use Have low number of

students Have single group that

cannot e divided Cannot have control

condition

Additional Options:• Add post-test to assess long-term change• Use alternative measures to minimize testing and

instrumentation effects

Two-Group, Pre-test/Post-test

Don’t Use Have single group of

students that cannot be divided

Use Have multiple groups

Additional Options:• Use random assignment to improve internal validity• Add post-test to assess long-term change• Use alternative measures to minimize testing and

instrumentation effects• Add additional conditions• Use covariates to improve internal validity and power

Within Participants Design

Don’t Use Early treatments affect

later treatments Early assessments affect

later assessments

Use Have low number of

students Have single group that

cannot be divided

Additional Options:• Add additional treatments• Counterbalance conditions to improve internal validity• Include pre-test to assess students before any treatment

Crossover DesignDon’t Use

First assessment, by itself, affects second

Have single group of students that cannot be divided

Use Have low number of

students Have multiple groups

Additional Options:• Include pre-test to assess before treatment• Add post-test to examine long-term change• Use random assignment to improve internal validity• Use alternative measures to minimize testing and

instrumentation effects

Interrupted Time-Series Design

Don’t Use Have only one session to

collect data Early assessments affect

later assessments

Use Have low number of

students Have single group that

cannot be divided Want to determine long-

term effects

Additional Options:• Add control condition to improve internal validity• Add additional treatment condition, with treatment at

different time to improve internal validity

More Complex Designs

0Use Multiple Treatments to Investigate Interactions (Interactions)

0Use Moderators to Determine When Treatment Has Effect (Concept of ATI)

0Use Mediators to Investigate How Treatment Has Effect (Mixed Method?)

Remember!0Each design has advantages and

disadvantages

0Often, there is no clear right way, although some designs will be better than others

0There is no single ideal study that eliminates all potential problems and all alternative hypotheses

0No one study can answer all of your questions!