1

Laura Martin

EDUTL 7645:

Test Construction Project

The Ohio State University

2

Table of Contents

Statement of the Problem.............................................................................................................. 1

Construct ...................................................................................................................................... 3

Content ......................................................................................................................................... 3

Content-Based Required Knowledge ................................................................................. 5

Language-Based Required Knowledge .............................................................................. 6

Operations ........................................................................................................................ 6

Types of Texts Produced ................................................................................................... 6

Addressees of Texts .......................................................................................................... 6

Length of Texts ................................................................................................................. 6

Topics ............................................................................................................................... 7

Readability........................................................................................................................ 7

Structural Range ............................................................................................................... 7

Vocabulary Range............................................................................................................. 7

Dialect/Accent/Style ......................................................................................................... 7

Speed of Processing .......................................................................................................... 7

Test Structure, Timing, Medium, and Techniques ........................................................................ 8

Test Structure.................................................................................................................... 8

Number of Items ............................................................................................................... 8

Number of Passages .......................................................................................................... 8

Timing .............................................................................................................................. 8

Medium/Channel .............................................................................................................. 9

Techniques........................................................................................................................ 0

Criterial Levels of Performance .................................................................................................... 9

Scoring Procedures ....................................................................................................................... 10

Opinion Paragraph Scoring Rubric .................................................................................... 11

Compare/Contrast Essay Scoring Rubric ........................................................................... 12

Validity of the Test Administered ................................................................................................. 13

Reliability of the Test Administered ............................................................................................. 13

Practicality of the Test Administered ............................................................................................ 14

Listening Script................................................................................................................. 15

Rationale ...................................................................................................................................... 17

References .................................................................................................................................... 21

1

Class: English as a Second Language (ESL) Social Studies Grade: 4th

Level: Ohio LEP Intermediate Level / TESOL Level 3 (Developing)

Topics: The U.S. Constitution; Comparing & Contrasting Vocabulary and Skills

STATEMENT OF THE PROBLEM

English Language Learners (ELLs) have the unique task of developing both language and

content-area skills simultaneously. It is essential that as ESL classes assist ELLs with language

development, they also promote grade-appropriate content understanding. In the state of Ohio,

learning standards are determined by the Ohio Department of Education, which has modified the

Ohio standards to fit the Common Core State Standards. The standards for Social Studies are set

forth in Ohio’s New Learning Standards: K-12 Social Studies (“Ohio’s New,” 2012). English

language standards for all students are described in the Common Core State Standards for

English Language Arts. Specific standards for ELLs are stated in both the Ohio English

Proficiency Standards for Limited English Proficient Students and the TESOL PreK-12 English

Language Proficiency Standards (“National Governors,” 2010b; “Ohio English,” 2012;

“TESOL”, 2006).

According to the Common Core State Standards, “teachers should recognize that it is

possible to achieve the standards… without manifesting native-like control of conventions and

vocabulary” (“National Governors,” 2010a, para. 2). Thus, because the fourth grade standards

for Social Studies in Ohio dictate that students must acquire understanding of the U.S.

Constitution, students in fourth-grade ESL Social Studies classrooms must also meet this

standard. By aligning learning objectives in this way, students are better prepared for the TLU

domain—a mainstream content-area classroom.

In this ESL Social Studies class, students will spend approximately three weeks on a unit

about the U.S. Constitution. They then must take a progress achievement test to: (a) demonstrate

what they have learned in this thematic unit and, (b) provide the teacher with feedback about

learner understanding and continued areas for necessary improvement.

Regarding content-area knowledge, this progress achievement test will measure student

understanding of key facts related to the U.S. Constitution. Regarding language development,

this test will measure learners’ abilities to analyze content knowledge, write passages that

compare and contrast branches of government, and write an opinion paragraph about First

Amendment rights. It will also measure the students’ abilities to listen and understand short

monologues in English related to the Social Studies curriculum.

Although this is not a high-stakes test, the results must be sufficiently detailed and

accurate so as to give both the teacher and learners an accurate picture of the students’ current

levels of understanding and how much progress was achieved as a result of this unit. These

results will provide the teacher with essential information that will likely result in positive

washback, as it can help her tailor the content of future lessons.

This test is limited to a 50-minute time constraint, as each ESL Social Studies class meets

for 55 minutes daily (and five minutes should be allotted for both routine classroom procedures

and the distribution/collection of exams). Although most students will complete the exam on a

2

pre-determined date, the test must be arranged in such a way that any students who happen to be

ill/absent that day can take the exam during a remedial period at a later date. Furthermore, as

much as possible, the test must be free of cultural bias, as the students in this class come from a

wide range of countries (e.g. Saudi Arabia, Somalia, El Salvador, Cuba, Vietnam, and China).

This unit and its corresponding progress achievement test must address the following

standards:

Ohio English Language Proficiency (ELP) Standards

Intermediate Level

Listening Standard 1.4 Comprehend the meaning of academic and/or specialized vocabulary

when spoken

Reading Standard 3.2 Identify the meaning of written vocabulary

Writing Standard 4.1 Write for varied purposes and audiences, with appropriate tone and voice,

using various media

Writing Standard 4.2 Write using a range of vocabulary, sentence structures, and verb tenses

TESOL PreK-12 English Language Proficiency Standards

Standard 2: English language learners communicate information, ideas, and concepts necessary

for academic success in the area of language arts.

Standard 5: English language learners communicate information, ideas, and concepts necessary

for academic success in the area of social studies.

Common Core State Standards for English Language Arts

Writing: Grade 4

W4.1: Write opinion pieces on topics or texts, supporting a point of view with reasons and

information.

W4.2: Write informative/explanatory texts to examine a topic and convey ideas and information

clearly.

Ohio’s New Learning Standards for K-12 Social Studies

Within the “Government Strand” of 4th-grade standards, students need to know:

Content Statement 19: The U.S. Constitution establishes a system of limited government and

protects citizens’ rights; five of these rights are addressed in the First Amendment.

Content Statement 20: A constitution is a written plan for government. Democratic

constitutions provide the framework for government in Ohio and the United States.

Content Statement 21: The Ohio Constitution and the U.S. Constitution separate the major

responsibilities of government among three branches.

Note: This unit is only designed to address the U.S. Constitution. Future lessons will draw on the

knowledge acquired in this unit to make connections between the U.S. and Ohio Constitutions.

3

CONSTRUCT

In this ESL Social Studies class, knowledge of the U.S. Constitution is defined as an

overall understanding of the basic history, purposes, and organization of the Constitution, and the

framework of government it provides for the United States.

English language ability in this context is defined as an ability to read, listen, write, and

speak about a variety of content-related topics. The manifestation of this is demonstrated through

the following abilities: (a) recognizing, understanding, and producing related key vocabulary; (b)

writing short sentences about content-specific information; (c) comparing and contrasting aspects

of this theme; and (d) sharing through writing an opinion about an issue related to the

Constitution.

CONTENT

Content-Based Required Knowledge

Historical Background of the Constitution

Americans formed a new national government under the Articles of Confederation in

1777. However, the Articles of Confederation were ineffective for running a new

nation.

Because of the ineffectiveness of the Articles of Confederation, early colonists held a

Constitutional Convention in 1787. Representatives from twelve states attended.

At the Constitutional Convention, colonists wrote the Constitution. It became an

official law in the United States on June 21, 1788 after 2/3 of the states voted for it.

Structure of the Constitution

The Constitution of the United States allowed for changes to be made to it when

necessary.

The first ten amendments are called the Bill of Rights.

The First Amendment to the Constitution provides the freedoms of religion, speech,

press, petition, and assembly.

o Freedom of Religion means that a person may choose his/her religion and how

or if he/she practices it.

o Freedom of Speech means that the government cannot prevent people from

saying what they want to or from expressing their beliefs, thoughts, and

opinions.

o Freedom of Press means that the government cannot control the news that is

reported to the public.

4

o Freedom of Assembly means that U.S. Americans can come together and join

groups without fear that the government will stop them.

o Freedom of Petition means that U.S. Americans can request changes in the

law or government.

The Constitution is organized into three major sections: (1) the Preamble; (2) the

Articles; and (3) the Amendments.

The Preamble is the introduction to the Constitution. It describes the purpose of the

Constitution.

There are seven articles in the Constitution. These articles establish how the

government will be organized and how the Constitution can be changed.

Amendments are changes made to the Constitution.

Collectively, the first 10 amendments are called the Bill of Rights.

Key Facts about the Constitution:

A constitution is a written document that describes the way a government is

organized and how its power is distributed.

The U.S. Constitution provides a framework for government, describing what it may

and may not do.

The Constitution protects the rights of citizens.

The Constitution limits the power of government.

The Constitution promotes the common good of all people.

Citizens of the United States elect government officials and representatives and make

decisions by voting. Thus, the U.S. people/citizens are the source of the government’s

authority and power.

Three Branches of Government:

The Constitution divides the power in the government between three different

branches so that one branch of government does not have too much power or control.

This system is called “separation of powers.”

The Legislative Branch makes laws. Officials in this branch of government can create

a proposal for a new bill, approve a bill, and pass it on to the executive branch.

o A law is a rule enforced by government that citizens must follow.

o The Senate and the House of Representatives are part of the Legislative Branch.

5

The Executive Branch carries out and enforces the laws. This branch has the power to

sign a bill and make it a law.

o The President of the United States is the head of the Executive Branch. He

gets help from the Vice President. The Executive Branch also includes the

Presidential Cabinet, government agencies, and the armed forces.

The Judicial Branch decides on the meaning of laws and applies the laws. This branch

has the power to determine whether or not a law follows the Constitution.

o The Supreme Court is part of the Judicial Branch.

All branches work together to uphold the Constitution. They all are based in

Washington, D.C., and they all carry out the responsibilities that are outlined for them

in the Constitution.

Language-Based Required Knowledge

Compare/Contrast Key Vocabulary

Contrast: to tell how two things are different

Compare: to tell how two things are alike or similar

Vocabulary to Describe Similarities

in the same way

both …and…

in addition

likewise

similarly

too

have in common

the same as

Vocabulary to Describe Differences

different from

compared with

unlike

but

conversely

in contrast

however

although

Key Question/Answer Formations

How are _________ and _________ alike?

o _________ and _________ are alike because….

How is _________ different from _________?

o _________ is different from _________ in its/their ….

How is _________ similar to _________?

o _________ is similar to _________ in its/their…..

What is a similarity between _________ and _________?

o One similarity between _________ and _________ is…

What is the difference between _________ and _________?

o One difference between_________ and _________ is….

6

Graphic Organizers

Venn diagrams are useful tools to show comparisons and contrasts graphically. To

complete a Venn diagram, draw two circles on the page. Write differences in the parts

of the circles that do not overlap and write similarities in the overlapping portion of

the circle.

Operations

Complete definitions (through gap-filling) of key vocabulary and concepts

Compare and contrast two of the three branches of government

State and support one’s opinion in writing about an aspect of the First Amendment

List the purposes of the U.S. Constitution

Identify and describe the major responsibilities of the branches of government

Organize information about the duties of the branches of government on a Venn diagram

Explain the significance of the first three words of the Constitution (“We the People…”)

Listen to a person describe his/her job and identify which branch of government he/she works for

Write short answers to content-specific questions about key ideas

Use a chart to sort useful “compare/contrast” vocabulary words and phrases into two categories

Types of Texts Produced

Gap-filling

Short answer responses

Opinion paragraph

Graphic organizer

2-paragraph compare/contrast essay

Addressees of Texts

Students are writing with awareness that the individual reading their text will be their

elementary ESL teacher.

The teacher, who is familiar with her students’ proficiency levels, will be the author of all

components of the test. In this way, she can ensure student comprehension is not affected

adversely by the test’s readability level or the types of tasks required of students.

Length of Texts

Gap-filling: 1-3 words per gap

Short answer: 1-2 sentences per question (sentence frames included to provide modification)

Opinion paragraph: 3-6 sentences

Compare/contrast essay: 2 paragraphs (6-12 sentences)

7

Topics

The U.S. Constitution (brief history of, structure, purpose)

Branches of government as outlined by the U.S. Constitution

Helpful vocabulary and techniques for compare/contrast writing

Readability

Texts throughout the exam are written at a level that is at or near a 4th-grade readability

level according to the Flesch-Kincaid Readability Score.

All parts of the test are written using Comic Sans (at least 12 pt. font), as this is the font

which most closely replicates print and is typically easiest for beginning ESL students to

understand (Epstein & Ormiston, 2007).

Structural Range

Structures included in the texts are high-frequency structures that will not impede

understanding for students of an intermediate level.

Vocabulary Range

Vocabulary utilized on the test consists of mostly high-frequency words. Some

exceptions are made for content-specific words that are related to this unit.

Dialect/Accent/Style

For the listening portion of the test, students listen to a pre-recorded track of three native

speakers of Standard American English.

On the writing portion of the exam, they will be expected to write short answers and

paragraphs in a simplified academic voice in a manner that is in keeping with the benchmarks

described in the Ohio English Language Proficiency Standards for intermediate-level learners.

Speed of Processing

The listening passages are reduced to a pace that is slower than a typical native speaker’s

speed of speech (around 70-80 words per minute).

8

TEST STRUCTURE, TIMING, MEDIUM, AND TECHNIQUES

Test Structure

Day 1

Part One: Listening Practice about Branches of Government

Part Two: Gap-Filling about Constitution/Key Terms

Part Three: Short Answer about the Constitution’s Purposes and Related Key Facts (sentence

frames provided for modification)

Part Four: Compare/Contrast Vocabulary Graphic Organizer Word Sort (word bank included)

Part Five: Opinion Paragraph about First Amendment Rights (visual support provided via

pictures of each right in order to assist student comprehension)

Day 2

Part One: Compare/Contrast Venn Diagram about Two Branches of Government (word bank

provided for extra support)

Part Two: Compare/Contrast Essay about Two Branches of Government

Number of Items

Day 1

Part One: Three questions; write which branch of government each speaker represents

Part Two: Seven gap-filling questions related to key vocabulary/concepts

Part Three: Three short answer questions about key concepts

Part Four: Ten words/phrases to “sort” into one of two columns

Part Five: One paragraph (3-6 sentences) sharing student’s opinion about First Amendment

Day 2

Part One: Test takers choose two of the three branches of government to compare/contrast in a

graphic organizer (Venn diagram)

Part Two: Two paragraphs about the similarities and differences between two branches of

government

Number of Passages

N/A

Timing

Day 1

Part One: 7 minutes

Parts Two, Three, Four, and Five: 43 minutes

Day 2

Parts One and Two: 50 minutes

9

Medium/Channel

Hand-written

Techniques

Technique Used Skills/Sub-skills Measured

Listening & Fill-in-the-Blank Response Listening; Recall duties of the Legislative,

Judicial, and Executive Branches

Short Answer (with sentence frames) Understanding of key concepts related to U.S.

Constitution

Gap-Filling Understanding key concepts related to U.S.

Constitution

Paragraph Writing Abilities to: (a) write for varied purposes; (b)

share opinion via writing; (c) compare/contrast

two main ideas

Graphic Organizer Word Sort Recognition of key compare/contrast

vocabulary words/phrases; Ability to

differentiate between similarities/differences

and various words used to talk about each

Venn Diagram Knowledge of how to sort items in a Venn

diagram; Ability to analyze two topics and

compare/contrast them; Understanding the

similarities/differences between the 3 branches

of government

CRITERIAL LEVELS OF PERFORMANCE

This test is criterion-referenced. The total number of points possible is 70. In order to

achieve an acceptable level, a student should obtain a score of at least 80% or a raw score of 56.

Students who score below this level will need to have a one-on-one meeting with the teacher to

address problem areas and set goals for future improvement.

10

SCORING PROCEDURES

On day one of the test, the total score possible is 50 points; on day two of the test, the

total score possible is 20 points. Scores from both days of the test will be added together to make

a total test value of 70 points.

The teacher will score all parts of the exam and will distribute results to students within

one week of the original test date. Students will receive their original exam papers and the

rubrics used to score both pieces of paragraph/essay writing.

Day 1

Each question in parts one (listening), two (gap-filling), and three (short answer) is worth

2 points. No partial credit will be given. Answers are based on content only and spelling does not

count against students unless meaning is impeded.

On part four of the exam (similarities/differences vocabulary chart), students get 1 point

for each word/phrase they correctly categorize in the graphic organizer.

Part five of the exam (opinion paragraph) is worth 10 points and is scored using the

analytic rubric on page 11.

Day 2

Part one (Venn diagram) is worth a total of 8 points. Students receive one point for listing

each of the names of the branches of government at the top of the diagram (for a total of 2

points). They receive one point for listing each of the two differences for both branches of

government (for a total of 4 points). Lastly, they receive one point for each of two similarities

between the branches of government (for a total of 2 points).

Part two (compare/contrast essay) is worth 12 points and is scored using the analytic

rubric on page 12.

11

Student: ____________________

Opinion Paragraph: First Amendment Rights

Scoring Rubric

0 1 2

Statement of

Opinion

Student does not give his/her opinion

about the most important 1st

Amendment right

Student gives his/her opinion about

the most important 1st Amendment

right

X

Understanding

of 1st

Amendment

Rights

It is clear that the student does not

understand the meaning of the 1st

Amendment right selected; or student

has not written about any of the 1st

Amendment rights

Student demonstrates that he/she has

at least a basic understanding of the

meaning of the 1st Amendment right

selected

X

Supporting

Reason #1

Student does not support his/her

opinion with a 1st reason

Student gives a 1st reason to support

his/her opinion that is unclear, illogical

and/or unrelated to the 1st Amendment

right selected

Student gives a 1st reason to support

his/her opinion that is mostly clear,

logical, and related to the 1st

Amendment right selected

Supporting

Reason #2

Student does not support his/her

opinion with a 2nd reason

Student gives a 2nd reason to support

his/her opinion that is unclear, illogical

and/or unrelated to the 1st Amendment

right selected

Student gives a 2nd reason to support

his/her opinion that is mostly clear,

logical, and related to the 1st

Amendment right selected

Supporting

Reason #3

Student does not support his/her

opinion with a 3rd reason

Student gives a 3rd reason to support

his/her opinion that is unclear, illogical

and/or unrelated to the 1st Amendment

right selected

Student gives a 3rd reason to support

his/her opinion that is mostly clear,

logical, and related to the 1st

Amendment right selected

Language Use

Student writing sample contains an

abundance of errors;

Meaning is incomprehensible

Student does not respond in paragraph

form/does not use complete sentences;

Several errors in spelling/grammar;

More than one error impedes meaning

Student responds in paragraph form

with complete sentences;

Few errors in spelling and grammar;

Errors do not impede meaning

Total Score: ________ / 10

12

Student: ______________________________

Compare/Contrast Essay: Two Branches of Government

Scoring Rubric

0 1 2 3

Content:

Differences

Does not include any

differences

Includes 1

difference with 0

errors in content;

OR includes 2

differences with

content errors in

both

Attempts to include

2 differences but

has a content error

in 1 of them

Includes 2

differences;

No errors in content

Content:

Similarities

Does not include any

similarities

Includes 1 similarity

with 0 errors in

content; OR includes 2

similarities with

content errors in

both

Attempts to include

2 similarities but

has a content error

in 1 of them

Includes 2

similarities;

No errors in content

Organization

Is not written in

paragraph form;

Is entirely

unorganized and

illogical;

No use of

transitions

Is not written in

paragraph form;

OR is in prose but

does not distinguish

between paragraphs;

Illogical

organization;

Poor transitions

Includes 2

paragraphs but does

not organize content

according to

similarities and

differences;

Transitions are

lacking or somewhat

confusing

Includes 2 distinct

paragraphs, using

1for similarities and

1 for differences;

Makes good

transitions between

sentences and

paragraphs

Language

Use

Language is

incomprehensible;

Full of spelling and

grammar errors;

Almost all errors

impede meaning;

Vocabulary is

extremely limited

Language is

somewhat

incomprehensible;

Several

spelling/grammar

errors that impede

meaning; Vocabulary

is very limited

Language is mostly

clear and

comprehensible;

Several

spelling/grammar

errors but they do

not impede meaning;

Vocabulary is limited

Language is clear

and comprehensible;

Few

spelling/grammar

errors are made;

Errors do not

impede meaning;

Utilizes a wide range

of vocabulary

Total Score: _____ /12

13

VALIDITY OF THE TEST ADMINISTERED

For this classroom-based progress achievement test, a high content validity is critical.

Because the test designer is also the classroom teacher, she will be able to guarantee high content

validity by ensuring that all topics addressed on the exam have been adequately covered in class.

Additionally, content validity is guaranteed because all test items correlate directly to

information that is included in the test specifications (Hughes, 2010).

It is also important that this test has a high face validity. As this is an ESL Social Studies

class, the test must address issues of both language and content. For this exam, the test designer

is able to maintain high face validity through the incorporation of both elements throughout the

exam. By dividing the test into sub-parts (five on the first day; 2 on the second day) with specific

focuses, the face validity is enhanced, as it becomes apparent that both language and content are

being tested.

Another validity-related concern is that content-specific segments of the test are, in fact,

assessing content (and not language) and vice versa. In order to address this concern, the test

designer has included several modifications. For example, in the short answer section of the

exam, the test designer has included sentence frames for students. In this way, the student is

supplied with all of the language he/she needs to demonstrate his/her knowledge. Additionally,

in the opinion paragraph segment of the test, pictures have been included to give students a

visual representation of each First Amendment right. On day two of the test, students are once

again provided with the language they need to demonstrate their knowledge through the

inclusion of a word/idea bank. In this way, students can focus their attention more on aspects of

compare/contrast essay writing and less on producing content-specific language.

RELIABILITY OF THE TEST ADMINISTERED

One way the test designer has addressed the reliability of this test is by ensuring that students

are “familiar with [the] format and testing techniques” (Hughes, 2010, p. 47). Because this three-

week unit will occur in the middle of the school year and after students have had several other

exams from the same teacher, it is safe to assume that students are already very familiar with the

testing procedures included in this test (listening questions, gap-filling, short answer, graphic

organizers, essay writing, etc.). Furthermore, prior to the exam date, students will have

participated in a review period, in which the teacher prepares them for the test by discussing the

basic format of this exam.

Another manner in which the reliability of this exam has been addressed is through the

provision of multiple samples of assessment (Hughes, 2010). Over the course of three weeks,

students will be assessed daily through in- and out-of-class assignments. Thus, upon completion

of this exam, the teacher will be able to compare students’ scores with their performances on

other assessments.

In the event that a student is absent on the first day of the exam and has to make up the

test on a later date, the test designer has taken measures to guarantee reliability in the listening

section of the exam by ensuring that all students have access to the same information and

14

materials. To do this, she has created a pre-recorded listening track that follows the script on

page 16.

The test designer has also attempted to ensure reliability by providing “clear directions”

and “unambiguous items” (Hughes, 2010, pp. 46-47). In an effort to ensure that questions are

clear, the test designer has solicited feedback from other colleagues.

Lastly, reliability of this exam is enhanced through the provision of an objective set of

scoring criteria. The majority of the exam has one clear, objective answer. For the more

subjective writing segments of the exam, analytic rubrics have been created to ensure intra-rater

reliability (Hughes, 2010).

PRACTICALITY OF THE TEST ADMINISTERED

This test is quite practical, as it does not require extensive time or resources. The entire

test is paper-based, and although color copies of some pages of the exam are required (to ensure

clarity), the small number (18) of students taking the exam makes this feasible.

Aside from copies of the test, the only other required materials are a pre-recorded audio

track for the listening section and lined pieces of paper on which the students will complete their

opinion paragraphs. The pre-recorded listening track will be recorded as an .mp3 file by the

teacher and three of her coworkers (to imitate three different voices) and can be done easily with

the technology included on a basic laptop.

The two-day timeframe for this test is reasonable, as it covers a three-week unit of class

and addresses both language and content standards. Although it is preferable for the tests to be

conducted on back-to-back days, it is possible for the second day of the exam to be completed

several days after the administration of the first day’s test if necessary.

15

ESL Social Studies Test

The U.S. Constitution

Part 1: Listening

Listening Script

(To be read at a pace of about 70-80 words per minute)

Narrator (Teacher’s voice):

This is the Listening part of your test. I will now read the directions. Follow along as I

read them.

Directions: Listening to people talk about their jobs. You will hear each person speak two

times. After each person speaks, I will read the question in bold to the class. To answer

the question, write the correct branch of government on the line.

Are there any questions? (Pause)

Let’s begin.

We will now begin question number one.

Speaker #1 (Female Voice):

My name is Sonia. I went to law school at Harvard Law School. Now, I am part of the

U.S. Supreme Court. It is my responsibility to decide if a law follows the Constitution or

not.

(Pause. Then repeat.)

Narrator (Teacher’s Voice):

Question #1: Which branch of government does Sonia work for? (Pause)

We will now move on to question number two.

Speaker #2 (Male Voice):

My name is Joe. I am the Vice President of the United States. I work in Washington D.C.

with the President. It is my responsibility to help carry out and enforce the laws in our

country.

(Pause. Then repeat.)

16

Narrator (Teacher’s Voice):

Question #2: Which branch of government does Joe work for? (Pause)

We will now move on to question number three.

Speaker #3 (Male Voice):

Hello! My name is Jim. I am from Ohio. Now, I work in Washington D.C. in the House

of Representatives. It is my duty to help make and pass laws for our country. It is an

important job. I enjoy being part of Congress.

(Pause. Then repeat.)

Narrator (Teacher’s Voice):

Question #3: Which branch of government does Jim work for? (Pause)

This is the end of the listening part of the test. You may now turn to page two and

complete the rest of the test.

17

RATIONALE

With my only formal teaching experience being in the field of EFL, ESL classrooms are

still a relatively “new world” to me. However, the time I have spent recently as a substitute

teacher in ESL classrooms has allowed me to obtain a better picture of what this environment is

like. I used these recent experiences as a starting point for my test construction project.

The intended audience I chose for this project is based on my current understanding of a

typical environment in an ESL classroom in Ohio. I chose elementary-aged learners and a

content-based assessment because: (a) I could model my “imaginary” classroom environment

after classes I have observed recently; and (b) I felt that it would be a significant learning

experience for me, having little formal teaching experience with such young learners and

content-based lessons.

Once I decided on my classroom environment and learners, I was then faced with the task

of determining the content on which I wanted to base my assessment. In order to do this, I

explored the Ohio Department of Education website and read the current English Language

Proficiency standards and the content standards for Social Studies and English Language Arts.

After finding some connections between the Social Studies and English Language Proficiency

standards, I decided to design an assessment based on a thematic unit about the U.S. Constitution

that utilized language as a tool to demonstrate understanding and analysis. Based on the

standards that I selected, I created a list of required content knowledge for students to learn

related to both language use and Social Studies.

With my list of required content knowledge complete, I then began to design a test to

assess my students’ progress and their understanding of this content. I found this to be far more

difficult than I originally anticipated. Designing a content-specific test that assessed students on

their knowledge of Social Studies and their ability to use language to listen, read, and write about

it was a difficult task! I repeatedly designed test items that I later deleted or modified because

they tested too many skills simultaneously. For example, some items that I intended to use to

assess student comprehension of the content area were ineffective because they required high

levels of reading or writing. Naturally, in an ESL Social Studies class, language will overlap with

content. However, my challenge was to assess content knowledge without allowing language to

be an unnecessary hurdle in the demonstration of this knowledge. Likewise, I sought to assess

language skills in a way that was not adversely affected by a limited content knowledge.

This complicated process of designing appropriate test items was quite recursive. I

frequently wrote and rewrote items to fit my objectives and modified my test construct to align

with my test items. Ultimately, I chose to include several different types of assessment (listening,

gap-filling, short answer, essay, graphic organizer, etc.) in an effort to obtain multiple samples

across a wide spectrum and give students many “fresh starts.” I also weighted each segment of

the test similarly, so that no student was “punished” for making multiple errors on one portion of

the test (whether due to lack of understanding of that content or merely to difficulty with a

specific format of assessment). However, the “content-specific” questions were weighted more

heavily than the “language-specific” portions of the test. This was intentional, as the class is a

content-based class where language is used as a tool to learn and demonstrate knowledge of

content-specific information.

My rationale for the readability, structural range, vocabulary range, speed of processing,

and dialect/accent was rooted in knowledge I have obtained from my graduate coursework and

interactions I have had with other ESL teachers. From these two sources, I have learned that

18

when testing ELLs in content areas, it is beneficial to limit the structural and vocabulary range to

high-frequency language features and words and keep the readability at or below the student’s

grade level so that language does not impede understanding. I chose to use native English

speakers with a Standard American accent for the listening passages because this dialect is most

representative of the students’ TLU domain (a mainstream classroom in an Ohio public school).

Although the speed of processing is slower than that which would be necessary in the TLU

domain, I made this modification in an effort to accommodate the specific needs of English

Language Learners.

The organization of my test items was another aspect of the test design process that was

recursive for me. I carefully debated in which order to put the various parts of the test and settled

on the current order for several reasons.

The test begins with the listening segment for practicality purposes. Since the whole class

must complete this part of the exam simultaneously, it is important to conduct this section first so

that students can use the remaining class time to complete the other parts of the test at their own

pace without being interrupted.

I debated between using either the gap-filling segment or the compare/contrast word sort

activity for the second part of the test. Initially, I considered including the compare/contrast word

sort in order to motivate the students, as this segment of the exam would most likely be easier for

most students than the gap-filling questions. However, in the end, I chose to place the gap-filling

questions in “part two” of the test because I felt that it was important to the test’s face validity

for the first non-listening part of the assessment to be directly related to Social Studies content.

Since the gap-filling exercise is the easiest set of content-based questions, I thought this was an

appropriate way to begin the written portion of the exam.

The opinion paragraph is the last question on the first day of the exam because I wanted

to ensure student awareness of the amount of time they have left to spend on paragraph writing. I

feared that if I included this item in the beginning of the test, students would spend too much

time writing and not have sufficient time to complete other portions of the test.

Lastly, for the second day of this test, I chose to require both a graphic organizer and an

essay because I thought that the Venn diagram would serve as a beneficial pre-writing tool for

students to refer to while writing their essays. Since they will often be required to do pre-writing

in the form of graphic organizers, webbing, or outlines in the TLU domain, it is useful for

students to also do pre-writing on this ESL Social Studies exam.

Another obstacle I encountered in my test design process was in determining the best way

to create a valid listening section. In one of the earliest versions of my test, I included visual

support for ELLs by providing pictures of each individual speaking. However, because I did not

want the questions to be inauthentic, I used only real people who are current government

officials. I later realized that the inclusion of the full names and pictures of these people posed a

threat to my test’s validity because it added an element of background knowledge that could not

be controlled. Although it was relatively unlikely, it was possible that some students could

recognize the names or photos of the government officials. If this occurred, students could

answer the questions without listening to the pre-recorded track and my test would no longer be

assessing listening skills. To address this, I removed the pictures and used only the first names of

three government officials (Sonia Maria Sotomayor, Joe Biden, and Jim Butler).

After creating several initial drafts of my test, I sent it to a currently-practicing teacher

who teaches 4th-grade ESL students in Iowa for feedback. She confirmed that the readability

level, types of test items, and length of test were appropriate for this level of learner. However,

19

she also suggested that I improve my test by adding visual support for students and making the

exam more aesthetically pleasing.

In response to her feedback, I continued to revise my test. I added more “white space,”

changed the font, and added pictures to several parts of the exam. For example, I added pictures

to represent each of the First Amendment rights in the opinion paragraph question in an effort to

provide students with a visual reminder of the meaning of each right and allow them to focus

their efforts more on writing and less on content. Likewise, on the second day of the test, I added

pictures to represent each of the branches of government. The pictures included are ones that

students would have seen repeatedly in class sessions and on lesson materials. I also added an

“idea bank” in which I listed the content that students would most likely want to include in their

Venn diagrams and compare/contrast essays. My motivation for doing this was to provide

students with the content-specific language they need to complete the task successfully. Since I

tested content in earlier parts of the exam, I did not feel that including this “idea bank” would

detract from the validity of the test or my ability to assess students’ content knowledge.

In addition to the 4th-grade ESL teacher in Iowa, I also sent my test to a 4

th-grade teacher

in North Carolina who works with low-income students. After receiving her feedback, I was

faced with a new challenge. She recommended that I add word banks for the gap-filling portion

of my exam to make it easier and also suggested that I make the test open-book. She claimed that

if the test were open-book, the students could work on their “finding proof” skills (a focus of the

Common Core State Standards for English Language Arts). After receiving her feedback, my

challenge was not to determine how to incorporate her suggestions, but whether to incorporate

her suggestions.

After much consideration, I chose not to use these two suggestions. I felt that if I were to

add a word bank for the gap-filling portion of the exam, I would no longer get an accurate idea of

my student’s content knowledge because they might be able to use “test wiseness” to determine

correct answers (based on the forms of words, singular/plural, beginning consonant v. vowel,

etc.). Additionally, I was more interested in observing students’ ability to produce content-

specific terms than their ability to simply recognize them from a list. In my opinion,

incorporating a word bank into this portion of the test would compromise my objectives.

Likewise, if I allowed the students to take this test as an open-book exam, it would change the

skills and abilities being tested. I would no longer be testing content knowledge, but rather the

ability to read from a text and apply the information to the exam.

Although this individual’s suggestions would be useful if applied to a different context, I

did not feel that they were applicable to my objectives and the situation as described in my

“statement of the problem.”

Throughout the development of this test construction project, I have addressed TESOL

Standard 4.a (“Issues of Assessment for English Language Learners”), which states that it is

necessary for candidates to, “[d]emonstrate understanding of various assessment issues as they

affect ELLs, such as accountability, bias, special education testing, language proficiency, and

accommodations in formal testing situations” in several ways (“TESOL,” 2010, p. 56).

First of all, I have addressed performance indicator 4.a.1, which states that candidates

should “[d]emonstrate an understanding of the purposes of assessment as they relate to ELLs and

use the results appropriately” (“TESOL,” 2010, p. 58). Although there are many different types

of assessments, I have chosen to create a progress achievement test for this test construction

project. Naturally, the purposes of this type of test and the ways in which I, the teacher, can use

the results vary considerably from other forms of assessment (i.e. diagnostic, proficiency, and

20

high-stakes tests). As I mentioned in my “statement of the problem,” I would hope to use this

assessment to provide me with information about my students’ understanding of the required

content knowledge. Additionally, I would use the results to inform future lessons in my ESL

Social Studies classroom. If few students achieved the criterial level of performance, it would

indicate to me that it was necessary to readdress specific aspects of the content so that my

students fully comprehend the information and can achieve the standards listed in my objectives.

In the development of my progress achievement test, I have also addressed performance

indicator 4.a.2, which states that candidates must be“[k]nowledgeable about and able to use a

variety of assessment procedures for ELLs” (“TESOL,” 2010, p. 58). I have designed a valid,

reliable, and practical exam that utilizes a variety of assessment techniques (listening, gap-filling,

short answer, graphic organizers, paragraphs, essays, etc.). Additionally, as is stated in my

description of the test’s reliability (pp. 13-14), I would also use formative assessments

accumulated throughout the course of the three-week unit to assess my students’ comprehension

and abilities related the aforementioned standards. By comparing students’ formative and

summative assessment results, I would obtain a clearer understanding of their progress.

Similarly, my discussion of this test’s validity, reliability, and practicality (pp. 13-14)

demonstrate that I am able to meet performance indicator 4.a.3, which states that candidates must

“[d]emonstrate an understanding of key indicators of good assessment instruments” (“TESOL,”

2010, p. 59). Through the recursive test design process, I have interacted with these three critical

characteristics of an assessment tool by evaluating my test in relation to each area. Likewise, I

have modified my assessment repeatedly in an effort to increase its validity, reliability, and

practicality.

Lastly, I have added multiple accommodations for ELLs to my exam in order to address

TESOL performance indicators 4.a.4 (candidates“[d]emonstrate an understanding of the

advantages and limitations of assessments including accommodations for ELLs”) and 4.a.5

(candidates “[d]istinguish among ELLs’ language differences, giftedness, and special education

needs”) (“TESOL,” 2010, p. 59). For example, in order to refrain from linguistic bias and to

address the linguistic limitations of the ELLs in my classroom, I have added “sentence frames”

to the short answer segment of this test. By doing this, I provided students with a large portion of

the language they need in order to answer the question. Likewise, recognizing that it would be

difficult for ELLs to produce extensive content-related vocabulary and then sort it in a graphic

organizer, I have included a word bank for students to use as a starting point for this part of the

test. Lastly, I have incorporated several pictures on the test in an effort to provide

accommodation via visual support for ELLs. Because I am aware that pictures are often a source

of cultural bias, the graphics I chose to use on my exam are culturally-sensitive and are ones that

my students would already be familiar with and would have seen in the context of in-class

worksheets and/or lesson materials.

The completion of this test construction project has been an eye-opening experience for

me. As a former EFL teacher, I created a multitude of performance achievement tests for my

English Language Learners. However, it was not until I began this project that I truly considered

elements of validity, reliability, practicality, and bias when creating an exam. Constructing a

good assessment tool is not something that can or should be done quickly. In addition, as I

worked on this project, I learned that assessments should not merely demonstrate student

learning, but should also contribute to student learning. Creating an assessment tool that meets

these criteria and those addressed in state standards is a difficult task. However, it is one that is

essential if I desire positive washback from my assessment.

21

References

Epstein, R. & Ormiston, M. (2007). Tools and Tips for Using ELT Materials. Ann Arbor, MI:

University of Michigan Press.

Hughes, A. (2010). Testing for language teachers (2nd

ed.). Cambridge: Cambridge University

Press.

National Governors Association Center for Best Practices & Council of Chief State School

Officers. (2010). Application of Common Core State Standards for English language

learners. Retrieved from http://www.corestandards.org/assets/application-for-english-

learners.pdf

National Governors Association Center for Best Practices & Council of Chief State School

Officers. (2010). English language arts standards: Writing: Grade 4. National Governors

Association Center for Best Practices & Council of Chief State School Officers:

Washington, DC.

Ohio English Language Proficiency (ELP) Standards. (2012). In Ohio Department of Education

Learning supports. Retrieved from http://education.ohio.gov/GD/Templates/Pages/ODE/

ODEDetail.aspx?page=3&TopicRelationID=500&ContentID=6621&Content=133989

Ohio's New Learning Standards: K-12 Social Studies. (2012). In Ohio Department of Education:

Teaching. Retrieved from http://www.ode.state.oh.us/GD/Templates/Pages/ODE/ODE

Detail.aspx?page=3&TopicRelationID=1706&ContentID=126270&Content=137752.

TESOL International Association. (2006). PreK-12 English language proficiency standards.

TESOL International Association: Alexandria, VA.

TESOL International Association. (2010). Standards for the recognition of initial TESOL

programs in P-12 ESL teacher education. TESOL International Association: Alexandria, VA.