Evaluate: 2007 version

55
1 Evaluate ! Evaluating School Media Services LIB 620 Library Management Fall 2009

description

 

Transcript of Evaluate: 2007 version

Page 1: Evaluate: 2007 version

1

Evaluate!

Evaluating School Media Services

LIB 620 Library ManagementFall 2009

Page 2: Evaluate: 2007 version

2

Preemptive Action:The busy librarian’s guide to

program evaluation

Based in part on:

A workshop developed by the Pennsylvania School Librarians Association

Professional Development Committee(No longer available online)

indicates the slide was originally from the above presentation

Page 3: Evaluate: 2007 version

3

Evaluate?

What do we mean by evaluate?– verb (used with object), -at ed, -at ing. ⋅ ⋅1. to determine or set the value or amount of;

appraise: to evaluate property.

2. to judge or determine the significance, worth, or quality of; assess: to evaluate the results of an experiment.

3. Mathematics. to ascertain the numerical value of (a function, relation, etc.).

• The American Heritage® Dictionary of the English Language, Fourth Edition © 2006 by Houghton Mifflin Company.

Page 4: Evaluate: 2007 version

4

What is evaluation?[1]

As defined by the American Evaluation Association: – evaluation involves assessing the strengths and

weaknesses of programs, policies, personnel, products, and organizations to improve their effectiveness.

– Evaluation is the systematic collection and analysis of data needed to make decisions• Evaluation Definition: What is Evaluation?

Page 5: Evaluate: 2007 version

5

What about school library media center evaluation?

Evaluation of school library media centers:–When you examine something or

someone, you examine the subject and make a judgment about the quality, significance, or condition of whatever is to be evaluated.• Emanuel T. Prostano and Joyce S. Prostano,

The School Library Media Center 5th ed. (Englewood, CO: Libraries Unlimited, 1999), 44.

Page 6: Evaluate: 2007 version

6

Purposes of evaluation

To find out what is right

To find out what is wrong

• Summary of definition in Blanche Woolls, The School Library Media Manager 2nd ed. (Westport, CT: Libraries Unlimited, 1999).

Page 7: Evaluate: 2007 version

7

What can you evaluate?

Everything!– Collection– Personnel– Library Layout– Programs– Curriculum collaboration

– Reference service/collection:• Interview skills• Quality of the answers

provided• Satisfaction of your

patrons– Students or teachers—or

other school staff

Page 8: Evaluate: 2007 version

8

What should you evaluate?

That depends . . . –On your priorities–On your economic needs and situation–On the policies and procedures established by

your school, school district and/or state–On how much time you have left over from

your other duties to devote to planning and executing an evaluation project

Page 9: Evaluate: 2007 version

9

Why evaluate?Because you have to

–Continuous assessment of services can be an ongoing requirement for accreditation/funding, etc.

Because you want to–You want to find out how well you’re doing–You want to find out how well others think

you’re doing–You want to find ways to improve your

service

Page 10: Evaluate: 2007 version

10

Why Evaluate?

• Schools are being evaluated by student academic achievement in reading, writing and math.

• Recent studies show student achievement correlates positively with effective school library programs.

Page 11: Evaluate: 2007 version

11

Why Evaluate?

Studies show the positive correlation with student achievement occurs when there is an effective school library program [and]:–School librarians are full time and–Librarians collaborate with teachers on

instructional planning.

Page 12: Evaluate: 2007 version

12

Why Evaluate?

Studies also show an effective school library program includes the following attributes: –Quality collections;– Increased hours of access beyond school day;–Professional development for teachers and

librarians;–Student access to technology; and–Collaboration with other types of libraries.

Page 13: Evaluate: 2007 version

13

Why Evaluate? Because. . .

To improve, the librarian must have baseline data about the program.

To be relevant, the librarian must know if the library services and resources are aligned with the school’s goals for student achievement.

To build a case for better support, the librarian needs data that demonstrates the value of the program.

Page 14: Evaluate: 2007 version

14

An evaluation enables you to:– Determine success in attaining program goals.– Determine students’ and teachers’ needs so they can

be incorporated into the program.– Provide a basis for resource allocation.– Recognize strengths and accomplishments.– Examine the impact of the program on

student learning.

What Can an Evaluation Do?

• Nancy Everhart, Evaluating the School Library Media Center, 1998)

Page 15: Evaluate: 2007 version

15

It makes sense—

Focus data collecting and evaluation on those components of a school library program that the research shows correlate with student achievement.

Page 16: Evaluate: 2007 version

16

Types of Evaluation

Formal

Informal

External

Internal

Formative During a program

Summative After the program is completed

Page 17: Evaluate: 2007 version

17

Evaluation requires standards

How do you know if the media center is “good”/ “not good” or even “good enough”?– “Good enough” for what? For whom?

Guidelines or rubrics or objectives for collections,

performance can be found in:– Beyond Proficiency: Achieving a Distinguished Library Media P

rogram. Kentucky Department of Education. August 2001

– Administering the Library Media Program• The library media specialist:

– Evaluates the Library Media Program through regular surveys for the purpose of enhancing services

Page 18: Evaluate: 2007 version

18

Types of Data

Quantitative Qualitative

Number of OPAC searches

Size of collection

Number of lessons planned with teachers

Students’ success rate in locating appropriate resources in OPAC searches

Collection supports the curriculum

Comprehensive and collaborative planning is in place

Page 19: Evaluate: 2007 version

19

Standards require interpretation

Quantitative: relatively easy– Library Media Staffing (Beyond Proficiency)

ENROLLMENT PROFICIENT DISTINGUISHED* ** * **

Under 200 1 0 1 .5200 - 500 1 .5 1 1501 - 800 1 1 1 1.5801 - 1200 1 1.5 1.5 1.51201 - 1600 1.5 1.5 2 21601 - 2000 2 2 2 22001 and Up 2.5 2 3 2* Certified Library Media Specialist**Library Clerk - Classified

Page 20: Evaluate: 2007 version

20

Standards require interpretation

Qualitative: more difficult–Program Evaluation Rubric, “

Beyond Proficiency,” pp. 20-26.–Standard 2:

• “The Library Media Program promotes and supports student learning and achievement through its policies, programs and collection.”

• Distinguished:–“Students are empowered to use the media

center to access information and reading for pleasure.”

Page 21: Evaluate: 2007 version

21

Interpretation means operationalization

Operationalize:–To define a concept in a way that can be

measured. In evaluation research, to translate program inputs, outputs, objectives, and goals into specific measurable variables. • Program Evaluation Glossary

http://www.epa.gov/evaluate/glossary/o-esd.htm

Page 22: Evaluate: 2007 version

22

Interpretation means creating goals and objectives

The purpose and study goals should determine the types of methods and measures you use to conduct the evaluation

Objectives will define your standard of excellence--the minimum level of appropriate service for your particular clientele – Jo Bell Whitlach, Evaluating Reference Services

Page 23: Evaluate: 2007 version

23

Methods of Evaluation

Obtrusive–People are aware of the evaluation

• Self-evaluation• Surveys• Observation

Unobtrusive–People are unaware of the evaluation

• Unobtrusive measures of physical facilities• Use of proxies—“mystery patrons”

Page 24: Evaluate: 2007 version

24Obtrusive reference evaluations in a school library context

Questionnaires or interviews of students or teachers

Numbers gathering:– Reference question counts– Numbers/types of reference books used– Circulation statistics

Observation– By external observer– Self-observation: Journal

Page 25: Evaluate: 2007 version

25

For example. . .

Research finding: Students whose school librarian plays an

instructional role tend to achieve higher than average test scores. This is also dependent on collaboration between school librarians and teachers and the inclusion of the library materials in the curriculum.

Page 26: Evaluate: 2007 version

26

Evaluation Model

1. Define the question.

2. Collect data--determine needed data and method of collection.

3. Analyze the data.

4. Formulate recommendations.

5. Develop an action plan.

Page 27: Evaluate: 2007 version

27

Research Finding Students whose school

librarian plays an instructional role tend to achieve higher than average scores.

What data are needed to determine whether or not the school librarian plays an instructional role in the school?

The Question

Evaluation Model Step 1.

Define the Question

Page 28: Evaluate: 2007 version

28

Look at the rubric

Beyond Proficiency:3. The Library Media program supportscollaborative planning with the staff forthe enhancement of instruction and support of student achievement.

Distinguished:

The LMS teams with teachers in the formal planning of student-centered authentic learning and project-based teaching and is a teaching partner.

Page 29: Evaluate: 2007 version

29According to

Information Power--

Instructional role is determined by:

Collaboration between school librarians and teachers that results in the inclusion of library materials in the curriculum.

Page 30: Evaluate: 2007 version

30According to

Information Power--

– Information literacy standards–Collaborative planning–Effective teaching

• Differentiated learning options• Inquiry• Assessment

–Student Engagement

Instructional role is defined by:

Page 31: Evaluate: 2007 version

31

According to Empowering Learners

Guideline 1:–The school library media program promotes

collaboration among members of the learning community and encourages learners to be independent, lifelong users and producers of information.

Page 32: Evaluate: 2007 version

32

Guideline 3:–The school library media program provides

instruction that addresses multiple literacies, including information literacy, media literacy, visual literacy, and technology literacy.

According to Empowering Learners

Page 33: Evaluate: 2007 version

33

Evaluation Model Step 2.

Collect data

Determine the type needed.–Quantitative/Qualitative

Determine the method of collection.–Existing statistic–Survey, questionnaire, focus group,

observation

Page 34: Evaluate: 2007 version

34

One method to collect data--

Use Information Power’s Teaching and Learning Rubric as a questionnaire.

Administer it to–School Library staff,–Administrative staff,–Teaching staff.

Summarize each group; determine where perceptions differ and agree.

Page 35: Evaluate: 2007 version

35Data collected from the Teaching and Learning Rubric are qualitative--

When completing the questionnaire--School librarian & principal: EXEMPLARY!Math teacher: 0 NOTHING!

Example:

Principal puts librarian on weekly faculty meeting agenda to encourage library use and promote curriculum-oriented library materials. Math teacher does not see an easy connect and tunes out.

Page 36: Evaluate: 2007 version

36

Evaluation Model Step 3.

Analyze the data

– Identify areas of agreement and disagreement among groups.

– Identify areas of strength and areas that need improvement.

– Identify areas where more data is needed.

How can the data be analyzed to determine whether or not the librarian plays an effective instructional role in the school?

Page 37: Evaluate: 2007 version

37

Agreement Example--

Target Indicator: Information Literacy Standards are integrated into content learning.

Library Staff Response

____Basic 100% Proficient ____ Exemplary

Administrative Response

____Basic 100% Proficient ____ Exemplary

Teaching Staff Response

12% Basic 88% Proficient ____ Exemplary

Page 38: Evaluate: 2007 version

38

Disagreement Example--

Target Indicator: Curriculum development is modeled and promoted.

Library Staff Response

100% Basic _____ Proficient ____ ExemplaryAdministrative Response

____Basic 100% Proficient ____ ExemplaryTeaching Staff Response

82%Basic 18% Proficient ____ Exemplary

Page 39: Evaluate: 2007 version

39

Strength Area Example--

Target Indicator: Collaborative planning is modeled and promoted.

Library Staff Response

____Basic 100% Proficient ____ Exemplary Administrative Response

____Basic 100% Proficient ____ Exemplary Teaching Staff Response

____Basic 100% Proficient ____ Exemplary

Page 40: Evaluate: 2007 version

40

Weakness Area Example--

Target Indicator: Students are engaged in reading, writing, speaking, viewing & listening for enjoyment, enrichment, & understanding.

Library Staff Response

____Basic 100% Proficient ____ Exemplary Administrative Response

100% Basic ____ Proficient ____ Exemplary Teaching Staff Response

82% Basic 18% Proficient ____ Exemplary

Page 41: Evaluate: 2007 version

41

Contradictory Data Example—Target Indicator:Effective teaching modeled &

promoted.Library Basic ____ Prof. 100% ExemplaryAdmin. ____ Basic ____ Prof. 100% ExemplaryTeacher Basic 60% Prof. 40% Exemplary

Target Indicator: Student achievement is assessed.Library 100% Basic Prof. ExemplaryAdmin. 100% Basic ____ Prof. ExemplaryTeacher 90% Basic 10% Prof. Exemplary

Page 42: Evaluate: 2007 version

42

Back to Step 2Collecting More Data – Focus Groups

1. How are student products and performances assessed in units involving use of the library?

2. How do you determine the products and performances to demonstrate mastery of content in units involving use of the library?

3. What is the role of reflection in student work involving use of the library?

Page 43: Evaluate: 2007 version

43

Evaluation Model Step 4.

Formulate recommendations

• Review the data, formulate recommendations.

• For example: – Because of the strong correlation between the school

librarian’s instructional role and student achievement, the Advisory Committee establishes a three year goal of reaching the Exemplary level in all target indicators in the Teaching and Learning Rubric

Role of an Advisory Committee of stakeholders—

Page 44: Evaluate: 2007 version

44Example:

Recommendations for First Year

– Review the Standards for the 21st Century Learner to ensure they align with content standards and set student expectations for analysis, evaluation and inquiry.

– Meet with teaching teams to solicit recommended revisions.

– Convene a workshop for new teachers to review the standards.

Standards for the 21st Century Learner

Adapted from

Page 45: Evaluate: 2007 version

45Example:

Recommendations for First Year

Collaborative Planning– Establish regular common planning time for

teachers with the library media staff.

Curriculum Development– Review school policies to remove any barriers that

prevent librarians from participating in building and district curriculum sessions.

Page 46: Evaluate: 2007 version

46Example:

Recommendations for First Year

Reading, Writing, Speaking, Viewing– Provide a series of author workshops and develop

plans with teachers for student reading, writing, speaking, and viewing responses to the author workshops.

– Involve Parents.

Effective Teaching– Clarify conflicting feedback through focus groups

targeting use of assessment, differentiation, & inquiry in instruction.

– Develop recommendations based on new data.

Page 47: Evaluate: 2007 version

47

Evaluation Model Step 5.

Develop an action plan

Library staff develop an action plan for each recommendation.

The Advisory Committee of Stakeholders reviews and approves the action plans.

Page 48: Evaluate: 2007 version

48

Action Plan Example:

Target Indicator: Curriculum development is modeled and promoted

Objective Activity Documentation Participants New Resources

Completion Date

Review barriers preventing librarians from participating in curriculum sessions.

Review policies

School Council minutes

School Council & admin.

None projected

Sept. 1

Secure schedule

Post schedule in lib. & faculty rm.

Principal None Sept. 10

Participate in mtgs.

Meeting minutes Librarians None projected

June 1

Report back

Faculty mtg. minutes

Librarians None Monthly

Written material

In professional library

Librarians None Within 5 days of mtg

Page 49: Evaluate: 2007 version

49

This example of an evaluation was…

• Type (which one?)– Formal– Internal– Formative

• Data collection methods:– Research– Rubric-based

questionnaire– Focus Group

• Data type:– Quantitative– Qualitative

Page 50: Evaluate: 2007 version

50

Unobtrusive evaluation in schools 1

Unobtrusive measures of physical facilities– “The basic premise . . . is that you can learn a great

deal . . . by looking at how things wear (‘erosion’), how things are left in the building (‘traces’) and how things are rearranged (‘adaptations for use’).”• Nancy Everhart, Evaluating the School Library

Media Center: Analysis Techniques and Research Practices. Libraries Unlimited, 1998.

Page 51: Evaluate: 2007 version

51

Unobtrusive evaluation in schools 2

Use of proxies—“mystery patrons”– Technique used more often in academic libraries and

government documents reference services• Half-right reference: the 55% rule• 5-minute rule

– Peter Hernon and Charles R. McClure, “Unobtrusive Reference Testing: The 55 Percent Rule” Library Journal April 15, 1986, 37-41.

• “It’s not true, and now we know why . . . the so-called “55% rule” has never been tested against a truly representative field sample.”

– John V. Richardson, Jr., “Reference Is Better Than We Thought,” Library Journal April 15, 2002, 41-42.

Page 52: Evaluate: 2007 version

52

Importance of a sophisticated model

Richardson:– “The reference service performance model [that led to

the 55% rule] was overly simplistic, samples were way too small, and the test questions were not representative of real-world reference questions.”

– “Students of reference service should learn about the existence of multiple performance outcomes (i.e., accuracy, utility, and satisfaction) and to recognize that each outcome is driven by different factors.”• “Reference Is Better Than We Thought.”

Page 53: Evaluate: 2007 version

53

Characteristics of good models

Measures of reference service must be– Valid

• They “accurately reflect the concept being studied.”– Reliable

• They “are stable and dependable, and provide consistent results with each repeated use.”

– Practical• They “require that data be relatively easy to collect.”

– Useful• They “provide information that can be used to

improve reference services.”– Whitlach, Evaluating Reference Services.

Page 54: Evaluate: 2007 version

54

Problems of Evaluation in School Context

Often the school library media specialist is the only one working in the library

An evaluation project can be time-consuming to plan and to put into action

Page 55: Evaluate: 2007 version

55

Solutions to evaluation problems

Depends on your local situation–Your resources:

• Time, money, available warm bodies

–You and your creativity

Make evaluation part of your routine–As you plan/prepare your program(s), include

an evaluation component• Appropriate, affordable, accessible, accountable