- DCWAC - Where we’ve come from, Where we are, Where we’re headed
NISE Network Evaluation: What We’ve Learned and Where We’re Headed
description
Transcript of NISE Network Evaluation: What We’ve Learned and Where We’re Headed
![Page 1: NISE Network Evaluation: What We’ve Learned and Where We’re Headed](https://reader035.fdocuments.us/reader035/viewer/2022081520/56816753550346895ddc051f/html5/thumbnails/1.jpg)
NISE Network Evaluation: What We’ve Learned and Where We’re
Headed
Network-Wide MeetingDecember 2012
![Page 2: NISE Network Evaluation: What We’ve Learned and Where We’re Headed](https://reader035.fdocuments.us/reader035/viewer/2022081520/56816753550346895ddc051f/html5/thumbnails/2.jpg)
Presenters • Elizabeth Kunz Kollmann,
Museum of Science, Boston, [email protected]
• Juli Goss, Museum of Science, Boston, [email protected]
• Liz Rosino, Oregon Museum of Science and Industry, [email protected]
• Gina Svarovsky, Science Museum of Minnesota, [email protected]
![Page 3: NISE Network Evaluation: What We’ve Learned and Where We’re Headed](https://reader035.fdocuments.us/reader035/viewer/2022081520/56816753550346895ddc051f/html5/thumbnails/3.jpg)
NISE Network Evaluation Team
Composed of evaluators from:
• Museum of Science, Boston
• Oregon Museum of Science and Industry
• Science Museum of Minnesota
![Page 4: NISE Network Evaluation: What We’ve Learned and Where We’re Headed](https://reader035.fdocuments.us/reader035/viewer/2022081520/56816753550346895ddc051f/html5/thumbnails/4.jpg)
Evaluation Team Structure
Summative Evaluation
Professional Impacts
Public Impacts
Formative Evaluation
Team-Based Inquiry
Annual Partner Survey / Data Mining
![Page 5: NISE Network Evaluation: What We’ve Learned and Where We’re Headed](https://reader035.fdocuments.us/reader035/viewer/2022081520/56816753550346895ddc051f/html5/thumbnails/5.jpg)
Network Communication Study
www.nisenet.org
![Page 6: NISE Network Evaluation: What We’ve Learned and Where We’re Headed](https://reader035.fdocuments.us/reader035/viewer/2022081520/56816753550346895ddc051f/html5/thumbnails/6.jpg)
6
Study purpose and focus
A study of communication within and between the tiers…Because this is a network, studying communication
highlights the ideas holding us all together.
![Page 7: NISE Network Evaluation: What We’ve Learned and Where We’re Headed](https://reader035.fdocuments.us/reader035/viewer/2022081520/56816753550346895ddc051f/html5/thumbnails/7.jpg)
7
Study purpose and focus
4 primary communication components:
NanoDays Regional hub structure
Face-to-face meetings nisenet.org
![Page 8: NISE Network Evaluation: What We’ve Learned and Where We’re Headed](https://reader035.fdocuments.us/reader035/viewer/2022081520/56816753550346895ddc051f/html5/thumbnails/8.jpg)
8
Methods
- Total of 24 professionals
Broad Reach PartnersTIER 3
Core PartnersTIER 1
Nano-infused PartnersTIER 2
Tier 1- 7 focus groups with
3-4 people each
![Page 9: NISE Network Evaluation: What We’ve Learned and Where We’re Headed](https://reader035.fdocuments.us/reader035/viewer/2022081520/56816753550346895ddc051f/html5/thumbnails/9.jpg)
9
Methods
- All participants considered “actively engaged”
- Variousmuseum types, sizes, and geographic region
Broad Reach PartnersTIER 3
Core PartnersTIER 1
Nano-infused PartnersTIER 2
Tiers 2 & 3- 39 individual interviews
![Page 10: NISE Network Evaluation: What We’ve Learned and Where We’re Headed](https://reader035.fdocuments.us/reader035/viewer/2022081520/56816753550346895ddc051f/html5/thumbnails/10.jpg)
10
FindingsThe communication that happens through various
Network components
![Page 11: NISE Network Evaluation: What We’ve Learned and Where We’re Headed](https://reader035.fdocuments.us/reader035/viewer/2022081520/56816753550346895ddc051f/html5/thumbnails/11.jpg)
11
NanoDays says…
![Page 12: NISE Network Evaluation: What We’ve Learned and Where We’re Headed](https://reader035.fdocuments.us/reader035/viewer/2022081520/56816753550346895ddc051f/html5/thumbnails/12.jpg)
12
NanoDays says…
You can do Nano!
![Page 13: NISE Network Evaluation: What We’ve Learned and Where We’re Headed](https://reader035.fdocuments.us/reader035/viewer/2022081520/56816753550346895ddc051f/html5/thumbnails/13.jpg)
13
NanoDays says…
You can do Nano!
Nano is
important!
![Page 14: NISE Network Evaluation: What We’ve Learned and Where We’re Headed](https://reader035.fdocuments.us/reader035/viewer/2022081520/56816753550346895ddc051f/html5/thumbnails/14.jpg)
14
NanoDays says…
You can do Nano!
Nano is
important!
Communication and dissemination of information is important!
![Page 15: NISE Network Evaluation: What We’ve Learned and Where We’re Headed](https://reader035.fdocuments.us/reader035/viewer/2022081520/56816753550346895ddc051f/html5/thumbnails/15.jpg)
15
Face-t0-face meetings say…
![Page 16: NISE Network Evaluation: What We’ve Learned and Where We’re Headed](https://reader035.fdocuments.us/reader035/viewer/2022081520/56816753550346895ddc051f/html5/thumbnails/16.jpg)
16
Face-t0-face meetings say…
We are here for you!
![Page 17: NISE Network Evaluation: What We’ve Learned and Where We’re Headed](https://reader035.fdocuments.us/reader035/viewer/2022081520/56816753550346895ddc051f/html5/thumbnails/17.jpg)
17
Face-t0-face meetings say…
We are here for you!
You are part of
a community!
![Page 18: NISE Network Evaluation: What We’ve Learned and Where We’re Headed](https://reader035.fdocuments.us/reader035/viewer/2022081520/56816753550346895ddc051f/html5/thumbnails/18.jpg)
18
Regional Hub Structure says…
![Page 19: NISE Network Evaluation: What We’ve Learned and Where We’re Headed](https://reader035.fdocuments.us/reader035/viewer/2022081520/56816753550346895ddc051f/html5/thumbnails/19.jpg)
19
Regional Hub Structure says…
We are your go-to resource!
![Page 20: NISE Network Evaluation: What We’ve Learned and Where We’re Headed](https://reader035.fdocuments.us/reader035/viewer/2022081520/56816753550346895ddc051f/html5/thumbnails/20.jpg)
20
Regional Hub Structure says…
We are your go-to resource!
You are also part of a local community!
![Page 21: NISE Network Evaluation: What We’ve Learned and Where We’re Headed](https://reader035.fdocuments.us/reader035/viewer/2022081520/56816753550346895ddc051f/html5/thumbnails/21.jpg)
21
Regional Hub Structure says…
We are your go-to resource!
You are also part of a local community!
Please contact
us!
![Page 22: NISE Network Evaluation: What We’ve Learned and Where We’re Headed](https://reader035.fdocuments.us/reader035/viewer/2022081520/56816753550346895ddc051f/html5/thumbnails/22.jpg)
22
Nisenet.org website says…
![Page 23: NISE Network Evaluation: What We’ve Learned and Where We’re Headed](https://reader035.fdocuments.us/reader035/viewer/2022081520/56816753550346895ddc051f/html5/thumbnails/23.jpg)
23
Nisenet.org website says…
You can do
MORE
nano!
![Page 24: NISE Network Evaluation: What We’ve Learned and Where We’re Headed](https://reader035.fdocuments.us/reader035/viewer/2022081520/56816753550346895ddc051f/html5/thumbnails/24.jpg)
24
Nisenet.org website says…
You can do
MORE
nano!
Here is a wealth of Network info and resources!
![Page 25: NISE Network Evaluation: What We’ve Learned and Where We’re Headed](https://reader035.fdocuments.us/reader035/viewer/2022081520/56816753550346895ddc051f/html5/thumbnails/25.jpg)
25
The Nano Bite and social networking says…
![Page 26: NISE Network Evaluation: What We’ve Learned and Where We’re Headed](https://reader035.fdocuments.us/reader035/viewer/2022081520/56816753550346895ddc051f/html5/thumbnails/26.jpg)
26
The Nano Bite and social networking says…
We have some interesting and timely information for you!
![Page 27: NISE Network Evaluation: What We’ve Learned and Where We’re Headed](https://reader035.fdocuments.us/reader035/viewer/2022081520/56816753550346895ddc051f/html5/thumbnails/27.jpg)
Communication that Network partners desire
Ideas, information, and practices that Network partners wished they heard more about.
27
![Page 28: NISE Network Evaluation: What We’ve Learned and Where We’re Headed](https://reader035.fdocuments.us/reader035/viewer/2022081520/56816753550346895ddc051f/html5/thumbnails/28.jpg)
How do I convey
challenging
content?
I guess it’s the SEI stuff, the societal and ethical implication stuff… I think it’s a bigger question that the field is struggling with.
What kind of information can we provide to the other partners to get them to feel comfortable with reaching out to diverse audiences?
28
Desired Communication
![Page 29: NISE Network Evaluation: What We’ve Learned and Where We’re Headed](https://reader035.fdocuments.us/reader035/viewer/2022081520/56816753550346895ddc051f/html5/thumbnails/29.jpg)
How are partners
engaging the
public in nano?
I would love to be able to know or gauge partners’ interest… have we created things that they really want to use?
One of the things that would be helpful for me, and I’m sure for other people, is to sort of have people’s experiences with these NanoDays kits and these activities somehow recorded.
29
Desired Communication
![Page 30: NISE Network Evaluation: What We’ve Learned and Where We’re Headed](https://reader035.fdocuments.us/reader035/viewer/2022081520/56816753550346895ddc051f/html5/thumbnails/30.jpg)
How can we communicate more?
I wish I knew more about the [NanoDays] evaluation, like what they’re finding out is more effective…. I’ve never heard them talk directly about the data that’s being collected.
I would just like to have more email, even if it’s just notices of new things coming out of whatever the national network is as well…maybe give me a menu of things to sign up for.
30
Desired Communication
![Page 31: NISE Network Evaluation: What We’ve Learned and Where We’re Headed](https://reader035.fdocuments.us/reader035/viewer/2022081520/56816753550346895ddc051f/html5/thumbnails/31.jpg)
Reflections
31
Directionality of NISE Network communication components
![Page 32: NISE Network Evaluation: What We’ve Learned and Where We’re Headed](https://reader035.fdocuments.us/reader035/viewer/2022081520/56816753550346895ddc051f/html5/thumbnails/32.jpg)
- NanoDays reports
- Educational product evaluation
- Network Communication Study
Learn more about evaluation
nisenet.org/catalog/eval
![Page 33: NISE Network Evaluation: What We’ve Learned and Where We’re Headed](https://reader035.fdocuments.us/reader035/viewer/2022081520/56816753550346895ddc051f/html5/thumbnails/33.jpg)
Learn more about evaluation
Evaluation of educational products
![Page 34: NISE Network Evaluation: What We’ve Learned and Where We’re Headed](https://reader035.fdocuments.us/reader035/viewer/2022081520/56816753550346895ddc051f/html5/thumbnails/34.jpg)
Learn more about the Network
Subscribe here!
The Nano Bitenisenet.org/community/nanobite
![Page 35: NISE Network Evaluation: What We’ve Learned and Where We’re Headed](https://reader035.fdocuments.us/reader035/viewer/2022081520/56816753550346895ddc051f/html5/thumbnails/35.jpg)
Learning more about the Network
Communitynisenet.org/community- Find your Regional Hub leader
and talk to a real person!
- Use the social networking links to connect with others in your region!
![Page 36: NISE Network Evaluation: What We’ve Learned and Where We’re Headed](https://reader035.fdocuments.us/reader035/viewer/2022081520/56816753550346895ddc051f/html5/thumbnails/36.jpg)
Annual Partner Survey
What was the purpose?
Consolidate requests for information about partners
Who did we want to hear from?
1,020 individuals considered ‘currently active’ in the Network
How did we collect the data?
Online survey in late October
Who responded?
296 individuals from 199 organizations
![Page 37: NISE Network Evaluation: What We’ve Learned and Where We’re Headed](https://reader035.fdocuments.us/reader035/viewer/2022081520/56816753550346895ddc051f/html5/thumbnails/37.jpg)
Professional ImpactsHow does the Network impact professionals’…
• Community
• Learning
• Use
3 years
Annual Partner Survey
Interviews
![Page 38: NISE Network Evaluation: What We’ve Learned and Where We’re Headed](https://reader035.fdocuments.us/reader035/viewer/2022081520/56816753550346895ddc051f/html5/thumbnails/38.jpg)
Public Impact of the Nano exhibition
www.nisenet.org
![Page 39: NISE Network Evaluation: What We’ve Learned and Where We’re Headed](https://reader035.fdocuments.us/reader035/viewer/2022081520/56816753550346895ddc051f/html5/thumbnails/39.jpg)
Nano mini-exhibition
400 sq. ft
Seven core components and seating area
70 identical copies going to Network Partners
![Page 40: NISE Network Evaluation: What We’ve Learned and Where We’re Headed](https://reader035.fdocuments.us/reader035/viewer/2022081520/56816753550346895ddc051f/html5/thumbnails/40.jpg)
Evaluation of Nano
Formative Evaluation• main focus: provide feedback for
improvement• Do individual pieces work for visitors?
Summative Evaluation• main focus: measure impact• What does the overall experience look like? • What does visitor learning look like?
![Page 41: NISE Network Evaluation: What We’ve Learned and Where We’re Headed](https://reader035.fdocuments.us/reader035/viewer/2022081520/56816753550346895ddc051f/html5/thumbnails/41.jpg)
Summative evaluation: Context
Five sites, variety of installations Science Museum of
MinnesotaDuluth Children’s Museum Science Spectrum (Lubbock, TX)Oregon Museum of Science & IndustryPort Discovery (Baltimore, MD)
Data collectedobservationssurveys and interviewscounting tallies
![Page 42: NISE Network Evaluation: What We’ve Learned and Where We’re Headed](https://reader035.fdocuments.us/reader035/viewer/2022081520/56816753550346895ddc051f/html5/thumbnails/42.jpg)
Data analysis
General Visitors
SMM (n=100)Duluth (n=103)
Port Discovery (n=32)OMSI (n=25)
Science Spectrum (n=16)
Spanish-Bilingual VisitorsOMSI (n=25)
Science Spectrum (n=22)
Visitors with Disabilities
Port Discovery Museum of Science
![Page 43: NISE Network Evaluation: What We’ve Learned and Where We’re Headed](https://reader035.fdocuments.us/reader035/viewer/2022081520/56816753550346895ddc051f/html5/thumbnails/43.jpg)
Early findings: Use
![Page 44: NISE Network Evaluation: What We’ve Learned and Where We’re Headed](https://reader035.fdocuments.us/reader035/viewer/2022081520/56816753550346895ddc051f/html5/thumbnails/44.jpg)
Early findings: Use
Visitors dig it!High visitor contact High dwell times
![Page 45: NISE Network Evaluation: What We’ve Learned and Where We’re Headed](https://reader035.fdocuments.us/reader035/viewer/2022081520/56816753550346895ddc051f/html5/thumbnails/45.jpg)
Early findings: Use
Visitors dig it!High visitor contact High dwell times
Visitor views of experienceInteresting and enjoyable“interactive”, “family-friendly”, “Informative”emphasis on connection of nano to daily life
![Page 46: NISE Network Evaluation: What We’ve Learned and Where We’re Headed](https://reader035.fdocuments.us/reader035/viewer/2022081520/56816753550346895ddc051f/html5/thumbnails/46.jpg)
Early findings: Learning
Visitors are learning!60% report increase in confidenceSimilar proportions of increase across levels of prior nano knowledge
![Page 47: NISE Network Evaluation: What We’ve Learned and Where We’re Headed](https://reader035.fdocuments.us/reader035/viewer/2022081520/56816753550346895ddc051f/html5/thumbnails/47.jpg)
Early findings: Learning
Visitors are learning!60% report increase in confidenceSimilar proportions of increase across levels of prior nano knowledge
NISE Net content map areasVisitor responses touched on all four areas of the content mapMost common: connection to daily lifeThis specific answer was more common
that general responses about nano or science
![Page 48: NISE Network Evaluation: What We’ve Learned and Where We’re Headed](https://reader035.fdocuments.us/reader035/viewer/2022081520/56816753550346895ddc051f/html5/thumbnails/48.jpg)
Emerging differencesgroup dwell times/visitor contactlevels of interest & enjoymentchanges in confidence
Institutional comparisons
![Page 49: NISE Network Evaluation: What We’ve Learned and Where We’re Headed](https://reader035.fdocuments.us/reader035/viewer/2022081520/56816753550346895ddc051f/html5/thumbnails/49.jpg)
Spanish-bilingual comparisonsprior nano knolwedgegroup dwell timesslightly different confidence gains
Spanish bilingual audience
![Page 50: NISE Network Evaluation: What We’ve Learned and Where We’re Headed](https://reader035.fdocuments.us/reader035/viewer/2022081520/56816753550346895ddc051f/html5/thumbnails/50.jpg)
Further analysis and next steps
Exploring other relationships between use and learning; demographic data
Additional analysis of bilingual data
Analysis of data from visitors with disabilities
Additional estimates for number of visitors who come in contact with Nano
Next Public Impacts study: Focus on NanoDays
![Page 51: NISE Network Evaluation: What We’ve Learned and Where We’re Headed](https://reader035.fdocuments.us/reader035/viewer/2022081520/56816753550346895ddc051f/html5/thumbnails/51.jpg)
Questions and Discussion?