Evaluate: Evaluating School Media Services

1

EvaluateEvaluate!!

Evaluating School Media ServicesEvaluating School Media Services

LIB 620 Library ManagementFall 2008

2

Preemptive ActionPreemptive Action::The busy librarian’s guide to

program evaluation

Based in part on:Based in part on:

A workshop developed by theA workshop developed by the Pennsylvania School Librarians Association

Professional Development CommitteeProfessional Development Committee((No longer available online)No longer available online)

3

Evaluate??

What do we mean by evaluation?What do we mean by evaluation?– TRANSITIVE VERB:

1. To ascertain or fix the value or worth of.

2. To examine and judge carefully; appraise. See synonyms at estimate.

3. Mathematics To calculate the numerical value of; express numerically.

• American Heritage® Dictionary of the English Language, Fourth

Edition. 2000 http://www.bartleby.com/61/

4

What is evaluation?

Evaluation aims to: Evaluation aims to: – find out how far aims and objectives have

been met– examine the outputs, outcomes and impacts

that have been produced– explore effectiveness of procedures and

delivery– assess cost-effectiveness and value-for-money

• Step-by-Step Guide to Evaluation

5

What about What about school library media center school library media center evaluation?evaluation?

Evaluation of Evaluation of school library media school library media centers:centers:– When you examine something or someone,

you examine the subject and make a judgment about the quality, significance, or condition of whatever is to be evaluated.

• Emanuel T. Prostano and Joyce S. Prostano, The School Library Media Center 5th ed. (Englewood, CO: Libraries Unlimited, 1999), 44.

6

Purposes of evaluationPurposes of evaluation

To find out what is rightTo find out what is right

To find out what is wrongTo find out what is wrong

• Blanche Woolls, The School Library Media Manager 2nd ed. (Westport, CT: Libraries Unlimited, 1999).

7

What can you evaluate?What can you evaluate?

Everything!Everything!– Collection– Personnel– Library Layout– Programs– Curriculum collaboration

– Reference service/collection:• Interview skills• Quality of the answers

provided• Satisfaction of your

patrons– Students or teachers—or

other school staff

8

What What shouldshould you evaluate? you evaluate?

That depends . . . That depends . . . – On your priorities

– On your economic needs and situation

– On the policies and procedures established by your school, school district and/or state

– On how much time you have left over from your other duties to devote to planning and executing an evaluation project

9

Why evaluate?Why evaluate?Because you have toBecause you have to

– Continuous assessment of services can be an ongoing requirement for accreditation/funding, etc.

Because you want toBecause you want to– You want to find out how well you’re doing– You want to find out how well others think

you’re doing– You want to find ways to improve your

service

10

Why Evaluate?Why Evaluate?

• Schools are being evaluated by student academic achievement in reading, writing and math.

• Recent studies show student achievement correlates positively with effective school library programs.

11


Studies show the positive correlation Studies show the positive correlation with student achievement occurs when with student achievement occurs when there is an effective school library there is an effective school library program:program:– School librarians are full time and

– Librarians collaborate with teachers on instructional planning.

12


Studies also show an effective Studies also show an effective school library program includes school library program includes the following attributes: the following attributes: – Quality collections;– Increased hours of access beyond school day;– Professional development for teachers and

librarians;– Student access to technology; and– Collaboration with other types of libraries.

13

Why Evaluate? Because. . .Why Evaluate? Because. . .

To improve, the librarian must have baseline To improve, the librarian must have baseline data about the program. data about the program.

To be relevant, the librarian must know if the To be relevant, the librarian must know if the library services and resources are aligned with library services and resources are aligned with the school’s goals for student achievement.the school’s goals for student achievement.

To build a case for better support, the To build a case for better support, the librarian needs data that demonstrates the librarian needs data that demonstrates the value of the program.value of the program.

14

What Can an Evaluation Do?What Can an Evaluation Do?

• Nancy Everhart, Evaluating the School Library Media Center, 1998)

An evaluation enables you to:An evaluation enables you to:– Determine success in attaining program goals.– Determine students’ and teachers’ needs so they can

be incorporated into the program.– Provide a basis for resource allocation.– Recognize strengths and accomplishments.– Examine the impact of the program on

student learning.

15

It makes sense—It makes sense—

Focus data collecting and Focus data collecting and evaluation on those components of a evaluation on those components of a school library program that the school library program that the research shows correlate with student research shows correlate with student achievement.achievement.

16

Types of EvaluationTypes of Evaluation

Formal

Informal

External

Internal

Formative

During a program

Summative

After the program is completed

17

Evaluation requires standardsEvaluation requires standards

How do you know if the media center is “good”/ “not How do you know if the media center is “good”/ “not good” or even “good enough”?good” or even “good enough”?– “Good enough” for what? For whom?

Guidelines or rubrics or objectives for collections, Guidelines or rubrics or objectives for collections,

performanceperformance– Beyond Proficiency: Achieving a Distinguished Library Media Pr

ogram. Kentucky Department of Education. August 2001

– Administering the Library Media Program

• The library media specialist:– Evaluates the Library Media Program through regular

surveys for the purpose of enhancing services

18

Types of DataTypes of Data

QuantitativeQuantitative QualitativeQualitative

Number of OPAC Number of OPAC searchessearches

Size of Size of collectioncollection

Number of lessons planned with teachers

Students’ success rate in Students’ success rate in locating appropriate locating appropriate resources in OPAC searchesresources in OPAC searches

Collection supports the Collection supports the curriculumcurriculum

Comprehensive and Comprehensive and collaborative planning is in collaborative planning is in placeplace

19

Standards require interpretationStandards require interpretation

Quantitative: relatively easyQuantitative: relatively easy– Library Media Staffing (Beyond Proficiency)

ENROLLMENT PROFICIENT DISTINGUISHED* ** * **

Under 200 1 0 1 .5200 - 500 1 .5 1 1501 - 800 1 1 1 1.5801 - 1200 1 1.5 1.5 1.51201 - 1600 1.5 1.5 2 21601 - 2000 2 2 2 22001 and Up 2.5 2 3 2* Certified Library Media Specialist**Library Clerk - Classified

20

Standards require interpretationStandards require interpretation

Qualitative: more difficultQualitative: more difficult– Program Evaluation Rubric, “

Beyond Proficiency,” pp. 20-26.– Standard 2:

• “The Library Media Program promotes and supports student learning and achievement through its policies, programs and collection.”

• Distinguished:– “Students are empowered to use the media

center to access information and reading for pleasure.”

21

Interpretation means operationalizationInterpretation means operationalization

Operationalize:Operationalize:– To define a concept in a way that can be

measured. In evaluation research, to translate program inputs, outputs, objectives, and goals into specific measurable variables. • Program Evaluation Glossary

http://www.epa.gov/evaluate/glossary/o-esd.htm

22Interpretation means creating Interpretation means creating goals and objectivesgoals and objectives

The purpose and study goals should determine The purpose and study goals should determine the types of methods and measures you use to the types of methods and measures you use to conduct the evaluationconduct the evaluation

Objectives will define your standard of Objectives will define your standard of excellence--the minimum level of appropriate excellence--the minimum level of appropriate service for your particular clientele service for your particular clientele – Jo Bell Whitlach, Evaluating Reference Services

23

Methods of EvaluationMethods of Evaluation

ObtrusiveObtrusive– People are aware of the evaluation

• Self-evaluation• Surveys• Observation

UnobtrusiveUnobtrusive– People are unaware of the evaluation

• Unobtrusive measures of physical facilities• Use of proxies—“mystery patrons”

24

Obtrusive reference evaluations in a school library contextObtrusive reference evaluations in a school library context

Questionnaires or interviews of students or Questionnaires or interviews of students or teachersteachers

Numbers gathering:Numbers gathering:– Reference question counts– Numbers/types of reference books used– Circulation statistics

ObservationObservation– By external observer– Self-observation: Journal

25

For example. . .For example. . .

Research finding:Research finding: Students whose school librarian plays an

instructional role tend to achieve higher than average test scores. This is also dependent on collaboration between school librarians and teachers and the inclusion of the library materials in the curriculum.

26

Evaluation ModelEvaluation Model

1.1. Define the question.Define the question.

2.2. Collect data--determine needed data and Collect data--determine needed data and method of collection.method of collection.

3.3. Analyze the data.Analyze the data.

4.4. Formulate recommendations.Formulate recommendations.

5.5. Develop an action plan.Develop an action plan.

27

Research FindingResearch Finding Students whose school

librarian plays an instructional role tend to achieve higher than average scores.

What data are needed to determine whether or not the school librarian plays an instructional role in the school?

The QuestionThe Question

Evaluation Model Step 1.

Define the Question

28

Look at the rubricLook at the rubric

Beyond Proficiency::3. The Library Media program supportscollaborative planning with the staff forthe enhancement of instruction and support of student achievement.

Distinguished:Distinguished:

The LMS teams with teachers in the formal planning of student-centered authentic learning and project-based teaching and is a teaching partner.

29According toAccording to

Information Power--Information Power--

Collaboration between school librarians and teachers that results in the inclusion of library materials in the curriculum.

30According toAccording to

Information Power--Information Power--

– Information literacy standards– Collaborative planning– Effective teaching

• Differentiated learning options• Inquiry• Assessment

– Student Engagement

Instructional role is defined Instructional role is defined by:by:

31

Evaluation Model Step 2.Evaluation Model Step 2.

Collect dataCollect data

Determine the type needed.Determine the type needed.– Quantitative/Qualitative

Determine the method of collection.Determine the method of collection.– Existing statistic

– Survey, questionnaire, focus group, observation

32

One method to collect data--One method to collect data--

Use Use Information Power’s Information Power’s Teaching and Teaching and Learning Rubric as a questionnaire.Learning Rubric as a questionnaire.

Administer it toAdminister it to– School Library staff,– Administrative staff,– Teaching staff.

Summarize each group; determine where Summarize each group; determine where perceptions differ and agree.perceptions differ and agree.

33Data collected from the Teaching and Learning Rubric are Data collected from the Teaching and Learning Rubric are qualitative--qualitative--

When completing the questionnaire--When completing the questionnaire--School librarian & principal: School librarian & principal: EXEMPLARY! EXEMPLARY!Math teacher: 0 NOTHING!Math teacher: 0 NOTHING!

Example:Example:

Principal puts librarian on weekly faculty Principal puts librarian on weekly faculty meeting agenda to encourage library use meeting agenda to encourage library use and promote curriculum-oriented library and promote curriculum-oriented library materials. Math teacher does not see an materials. Math teacher does not see an easy connect and tunes out.easy connect and tunes out.

34


Analyze the dataAnalyze the data

– Identify areas of agreement and disagreement among groups.

– Identify areas of strength and areas that need improvement.

– Identify areas where more data is needed.

How can the data be analyzed to determine How can the data be analyzed to determine whether or not the librarian plays an effective whether or not the librarian plays an effective instructional role in the school?instructional role in the school?

35

Agreement Example--Agreement Example--

Target Indicator: Information Literacy Standards Target Indicator: Information Literacy Standards are integrated into content learningare integrated into content learning..

Library Staff ResponseLibrary Staff Response

____Basic ____Basic 100%100% Proficient ____ Exemplary Proficient ____ Exemplary

Administrative ResponseAdministrative Response

____Basic ____Basic 100%100% Proficient ____ Exemplary Proficient ____ Exemplary

Teaching Staff ResponseTeaching Staff Response

12%12% Basic Basic 88% 88% Proficient ____ Exemplary Proficient ____ Exemplary

36

Disagreement Example--Disagreement Example--

Target Indicator: Curriculum development is Target Indicator: Curriculum development is modeled and promoted.modeled and promoted.

Library Staff Response Library Staff Response

100%100% Basic _____ Proficient ____ Exemplary Basic _____ Proficient ____ ExemplaryAdministrative ResponseAdministrative Response

____Basic ____Basic 100%100% Proficient ____ Exemplary Proficient ____ ExemplaryTeaching Staff ResponseTeaching Staff Response

82%82%Basic Basic 18% 18% Proficient ____ Exemplary Proficient ____ Exemplary

37

Strength Area Example--Strength Area Example--

Target Indicator: Collaborative planning is Target Indicator: Collaborative planning is modeled and promoted.modeled and promoted.


____Basic ____Basic 100%100% Proficient ____ Exemplary Proficient ____ Exemplary Administrative ResponseAdministrative Response

____Basic ____Basic 100%100% Proficient ____ Exemplary Proficient ____ Exemplary Teaching Staff ResponseTeaching Staff Response

____Basic ____Basic 100% 100% Proficient ____ Exemplary Proficient ____ Exemplary

38

Weakness Area Example--Weakness Area Example--

Target Indicator: Students are engaged in Target Indicator: Students are engaged in reading, writing, speaking, viewing & listening reading, writing, speaking, viewing & listening for enjoyment, enrichment, & understanding.for enjoyment, enrichment, & understanding.


____Basic ____Basic 100%100% Proficient ____ Exemplary Proficient ____ Exemplary Administrative ResponseAdministrative Response

100%100% Basic ____ Proficient ____ Exemplary Basic ____ Proficient ____ Exemplary Teaching Staff Response Teaching Staff Response

82%82% Basic Basic 18% 18% Proficient ____ Exemplary Proficient ____ Exemplary

39

Contradictory Data Example—Contradictory Data Example—Target Indicator:Effective teaching modeled & Target Indicator:Effective teaching modeled &

promoted.promoted.LibraryLibrary Basic ____ Prof. Basic ____ Prof. 100%100% Exemplary Exemplary

Admin.Admin. ____ Basic ____ Prof. ____ Basic ____ Prof. 100%100% Exemplary ExemplaryTeacherTeacher Basic Basic 60% 60% Prof. Prof. 40% 40% Exemplary Exemplary

Target Indicator: Student achievement is Target Indicator: Student achievement is assessed.assessed.LibraryLibrary 100%100% Basic Basic Prof. Prof. ExemplaryExemplary

Admin.Admin. 100%100% Basic ____ Prof. Basic ____ Prof. ExemplaryExemplaryTeacherTeacher 90% 90% Basic Basic 10% 10% Prof. Prof. ExemplaryExemplary

40

Back to Step 2Back to Step 2Collecting More Data – Focus GroupsCollecting More Data – Focus Groups

1.1. How are student products and performances assessed How are student products and performances assessed in units involving use of the library?in units involving use of the library?

2.2. How do you determine the products and performances How do you determine the products and performances to demonstrate mastery of content in units involving to demonstrate mastery of content in units involving use of the library?use of the library?

3.3. What is the role of reflection in student work What is the role of reflection in student work involving use of the library?involving use of the library?

41


Formulate recommendationsFormulate recommendations

• Review the data, formulate recommendations.Review the data, formulate recommendations.

• For example: For example: – Because of the strong correlation between the school

librarian’s instructional role and student achievement, the Advisory Committee establishes a three year goal of reaching the Exemplary level in all target indicators in the Teaching and Learning Rubric

Role of an Advisory Committee of stakeholdersRole of an Advisory Committee of stakeholders——

42Example:Example:

Recommendations for First YearRecommendations for First Year

– Review the Information Literacy Standards to ensure they align with content standards and set student expectations for analysis, evaluation and inquiry.

– Meet with teaching teams to solicit recommended revisions.

– Convene a workshop for new teachers to review the standards.

Information Literacy StandardsInformation Literacy Standards

43Example:Example:


Collaborative PlanningCollaborative Planning– Establish regular common planning time for

teachers with the library media staff.

Curriculum DevelopmentCurriculum Development– Review school policies to remove any barriers that

prevent librarians from participating in building and district curriculum sessions.

44Example:Example:


Reading, Writing, Speaking, ViewingReading, Writing, Speaking, Viewing– Provide a series of author workshops and develop

plans with teachers for student reading, writing, speaking, and viewing responses to the author workshops.

– Involve Parents.

Effective TeachingEffective Teaching– Clarify conflicting feedback through focus groups

targeting use of assessment, differentiation, & inquiry in instruction.

– Develop recommendations based on new data.

45

Evaluation Model Step 5Evaluation Model Step 5..

Develop an action planDevelop an action plan

Library staff develop an action plan for Library staff develop an action plan for each recommendation.each recommendation.

The Advisory Committee of The Advisory Committee of Stakeholders reviews and approves the Stakeholders reviews and approves the action plans.action plans.

46

Action Plan Example:Action Plan Example:

Target Indicator: Curriculum development is modeled and promotedTarget Indicator: Curriculum development is modeled and promoted

ObjectiveObjective ActivityActivity DocumentationDocumentation ParticipantsParticipants New New ResourcesResources

Completion Completion DateDate

Review barriers Review barriers preventing preventing librarians from librarians from participating in participating in curriculum curriculum sessions.sessions.

Review Review policiespolicies

School Council School Council minutesminutes

School School Council & Council & admin.admin.

None None projectedprojected

Sept. 1Sept. 1

Secure Secure scheduleschedule

Post schedule in Post schedule in lib. & faculty rm.lib. & faculty rm.

PrincipalPrincipal NoneNone Sept. 10Sept. 10

Participate Participate in mtgs.in mtgs.

Meeting minutesMeeting minutes LibrariansLibrarians None None projectedprojected

June 1June 1

Report Report backback

Faculty mtg. Faculty mtg. minutesminutes

LibrariansLibrarians NoneNone MonthlyMonthly

Written Written materialmaterial

In professional In professional librarylibrary

LibrariansLibrarians NoneNone Within 5 Within 5 days of mtgdays of mtg

47

This example of an evaluation was…This example of an evaluation was…

• TypeType– Formal– Internal– Formative

• Data collection methods:Data collection methods:– Research– Rubric-based

questionnaire– Focus Group

• Data type:Data type:– Quantitative– Qualitative

48

Unobtrusive evaluation in schools 1Unobtrusive evaluation in schools 1

Unobtrusive measures of physical facilitiesUnobtrusive measures of physical facilities– “The basic premise . . . is that you can learn a great

deal . . . by looking at how things wear (‘erosion’), how things are left in the building (‘traces’) and how things are rearranged (‘adaptations for use’).”

• Nancy Everhart, Evaluating the School Library Media Center: Analysis Techniques and Research Practices. Libraries Unlimited, 1998.

49

Unobtrusive evaluation in schools 2Unobtrusive evaluation in schools 2

Use of proxies—“mystery patrons”Use of proxies—“mystery patrons”– Technique used more often in academic libraries and

government documents reference services• Half-right reference: the 55% rule• 5-minute rule

– Peter Hernon and Charles R. McClure, “Unobtrusive Reference Testing: The 55 Percent Rule” Library Journal April 15, 1986, 37-41.

• “It’s not true, and now we know why . . . the so-called “55% rule” has never been tested against a truly representative field sample.”

– John V. Richardson, Jr., “Reference Is Better Than We Thought,” Library Journal April 15, 2002, 41-42.

50

Importance of a sophisticated modelImportance of a sophisticated model

Richardson:Richardson:– “The reference service performance model [that led to

the 55% rule] was overly simplistic, samples were way too small, and the test questions were not representative of real-world reference questions.”

– “Students of reference service should learn about the existence of multiple performance outcomes (i.e., accuracy, utility, and satisfaction) and to recognize that each outcome is driven by different factors.”• “Reference Is Better Than We Thought.”

51

Characteristics of good modelsCharacteristics of good models

Measures of reference service must beMeasures of reference service must be– Valid

• They “accurately reflect the concept being studied.”– Reliable

• They “are stable and dependable, and provide consistent results with each repeated use.”

– Practical• They “require that data be relatively easy to collect.”

– Useful• They “provide information that can be used to

improve reference services.”– Whitlach, Evaluating Reference Services.

52

Problems of Evaluation in School ContextProblems of Evaluation in School Context

Often the school library media specialist Often the school library media specialist is the only one working in the libraryis the only one working in the library

An evaluation project can be time-An evaluation project can be time-consuming to plan and to put into actionconsuming to plan and to put into action

53

Solutions to evaluation problemsSolutions to evaluation problems

Depends on your local situationDepends on your local situation– Your resources:

• Time, money, available warm bodies

– You and your creativity

Make evaluation part of your routineMake evaluation part of your routine– As you plan/prepare your program(s), include

an evaluation component• Appropriate, affordable, accessible, accountable

Evaluate: Evaluating School Media Services

Education

Transcript of Evaluate: Evaluating School Media Services