An analysis of evaluation activities planned in NN/LM outreach subcontracts 2001-2006

Mellanye Lackey2nd year Associate FellowNational Library of MedicineUNC-Chapel Hill Health Sciences Librarymjlackey@email.unc.edu

2

What I’ll talk about today

Gaining background knowledge Literature review and conversations w. experts Barriers

Project data Collection methods Analysis Results and graphs

Opportunities to learn more

3

Barriers to doing outreach evaluationPrior experiences

usefulness motive fear negative results

Getting buy-in relationships cooperation from partners support from authority appreciation of evaluation

Lack of time money knowledge about

evaluation processes

-Taut, S. M., & Alkin, M. C. (2003). Program staff perceptions of barriers to evaluation implementation. American Journal of Evaluation. 24, 213-226.

4

5

Data gathering

6

Collecting project data

Projects includedPublic Health Information ModelAccess to Electronic Health Information

SubcontractConsumer Health Initiative SubcontractResource Library Outreach Subcontract Health Professional Information Outreach

7

Project data

Funding from $2,000 to $70,000 reporting evaluation plan

Reviewed >500 records (1/2 of all NN/LM projects)

n = 144

8

Data collection

Methods

QuestionnaireTally/LogPre/Post dataInterview/AnecdoteEvaluation formFocus groupObservation/Usability study

Approaches

Evidence for future useInternal committeeNeeds assessmentSocial capitalCase study

10

Evaluation methods/approaches planned(n=144, Number of methods>total subcontracts b/c of multiple methods planned)

61

37

29

1916

8

27

18

42 2

77

0

10

20

30

40

50

60

70

80Q

uest

ionn

aire

Tally

/Log

P

re/P

ost d

ata

Inte

rvie

w/ A

necd

ote

Eva

luat

ion

form

Focu

s gr

oup

Obs

erva

tion/

Usa

bilit

y st

udy

Evi

denc

e fo

r fut

ure

use

Nee

ds A

sses

smen

tS

ocia

l cap

ital

Cas

e st

udy

Inte

rnal

com

mitt

ee

Method / approach

Nu

mb

er

of

su

bc

on

tra

cts

11

Evaluation methods/appraoches vs. funding levels

0

10,000

20,000

30,000

40,000

50,000

60,000

70,000

0 1 2 3 4 5 6 7 8 9 10 11 12 13

Evaluation methods/approaches

Fu

nd

ing

le

ve

ls

Questionnaire (77)

Tally/Log (61)

Pre/Post data (37)

Interview/Anecdote (29)

Evaluation form (19)

Focus group (16)

Observation/Usability study(8)Case study (2)

Future Use (27)

Needs Assessment (18)

Social capital (4)

Internal committee (2)

77 8 2 27 18 4 229 16193761

12

Number of outreach subcontracts planning 1-6 methods (n=144)

5048

34

10

1 10

5

10

15

20

25

30

35

40

45

50

1 method 2 methods 3 methods 4 methods 5 methods 6 methods

Number of methods

Nu

mb

er

of

su

bc

on

tra

cts

$2500-$50,019

$2000-$56,000

$3200-$50,000

$2500-$50,000

$17,973 $40,000

13

Limits of the data

Records created at project inception Data sometimes rich, more often brief

How many participants? Focus groups? How much time? Terms vague (eg. evaluation form)

Growing emphasis on evaluation with Outreach Evaluation Resource Center (OERC)

14

Further research ideas

Compare evaluation planned with final report

Get more specifics about each plan Examine formative vs. summative Collect information about “approaches” How get more data?

15

Conclusions

At all funding levels: Intent to evaluate Variety of methods

Opportunities to learn more Bibliography of resources

http://www.hsl.unc.edu/AboutLib/Staff/LiaisonWebPages/PHliaison.cfm

Outreach Evaluation Resource Center

http://nnlm.gov/evaluation/index.html