Designing a “Point of Selection” Study...resource selection Methodic review of 483 potential...
Transcript of Designing a “Point of Selection” Study...resource selection Methodic review of 483 potential...
![Page 1: Designing a “Point of Selection” Study...resource selection Methodic review of 483 potential resources to establish the 65 unique resources that would be used in the simulations.](https://reader033.fdocuments.us/reader033/viewer/2022041610/5e37796c21712a142736da2e/html5/thumbnails/1.jpg)
Designing a “Point of Selection” Study Using Simulations:
From Trailhead to Terminus
@RSICStudyLG-81-15-0155
![Page 2: Designing a “Point of Selection” Study...resource selection Methodic review of 483 potential resources to establish the 65 unique resources that would be used in the simulations.](https://reader033.fdocuments.us/reader033/viewer/2022041610/5e37796c21712a142736da2e/html5/thumbnails/2.jpg)
![Page 3: Designing a “Point of Selection” Study...resource selection Methodic review of 483 potential resources to establish the 65 unique resources that would be used in the simulations.](https://reader033.fdocuments.us/reader033/viewer/2022041610/5e37796c21712a142736da2e/html5/thumbnails/3.jpg)
presenters:● Amy Buhler (University of Florida – University Librarian)
[email protected]● Joyce Kasman Valenza, PhD (Rutgers University – Assistant
Teaching Professor) [email protected]@joycevalenza
● Brittany Brannon (OCLC Research – Research Support Specialist) [email protected]
@RSICStudy
![Page 4: Designing a “Point of Selection” Study...resource selection Methodic review of 483 potential resources to establish the 65 unique resources that would be used in the simulations.](https://reader033.fdocuments.us/reader033/viewer/2022041610/5e37796c21712a142736da2e/html5/thumbnails/4.jpg)
our early trekkers
![Page 5: Designing a “Point of Selection” Study...resource selection Methodic review of 483 potential resources to establish the 65 unique resources that would be used in the simulations.](https://reader033.fdocuments.us/reader033/viewer/2022041610/5e37796c21712a142736da2e/html5/thumbnails/5.jpg)
important links:
RSIC LibGuide: https://guides.uflib.ufl.edu/RSIC
Resource Hyperdoc: http://tinyurl.com/rsictrailhead
@RSICStudy
![Page 6: Designing a “Point of Selection” Study...resource selection Methodic review of 483 potential resources to establish the 65 unique resources that would be used in the simulations.](https://reader033.fdocuments.us/reader033/viewer/2022041610/5e37796c21712a142736da2e/html5/thumbnails/6.jpg)
introduction: basecamp
@RSICStudy
![Page 7: Designing a “Point of Selection” Study...resource selection Methodic review of 483 potential resources to establish the 65 unique resources that would be used in the simulations.](https://reader033.fdocuments.us/reader033/viewer/2022041610/5e37796c21712a142736da2e/html5/thumbnails/7.jpg)
overviewOur study examines students’ “point of selection” behavior (i.e. the point at which an information seeker determines a resource meets their information need) and their judgements associated with the helpfulness, citability, and credibility of various online information resources. It also assessed students’ ability to identify the container(i.e. document type) in which the information is housed.
@RSICStudy
![Page 8: Designing a “Point of Selection” Study...resource selection Methodic review of 483 potential resources to establish the 65 unique resources that would be used in the simulations.](https://reader033.fdocuments.us/reader033/viewer/2022041610/5e37796c21712a142736da2e/html5/thumbnails/8.jpg)
research questions1. How do STEM students determine the credibility of online information resources?
○ What cues from a web search results screen do students use to judge the credibility of online information resources?
○ How do students’ characteristics influence their credibility judgments of online information resources?
2. Do STEM students differentiate among different types of online information resourcesduring point of selection?○ What cues from a web search results screen do students use to identify online
information resources?○ How do students’ characteristics and experiences influence their identification behavior?
@RSICStudy
![Page 9: Designing a “Point of Selection” Study...resource selection Methodic review of 483 potential resources to establish the 65 unique resources that would be used in the simulations.](https://reader033.fdocuments.us/reader033/viewer/2022041610/5e37796c21712a142736da2e/html5/thumbnails/9.jpg)
Self-reported instruments
Simulation
Data Cleaning & Quality Control
Codebook Creation
Takeaways
Q & A
Today’s trek
@RSICStudy
![Page 10: Designing a “Point of Selection” Study...resource selection Methodic review of 483 potential resources to establish the 65 unique resources that would be used in the simulations.](https://reader033.fdocuments.us/reader033/viewer/2022041610/5e37796c21712a142736da2e/html5/thumbnails/10.jpg)
what was in our research pack?
Prescreen Survey
Interview Questions
Simulation of SERPs (not to be confused with sherpas!)
Think-Aloud Protocols
@RSICStudy
![Page 11: Designing a “Point of Selection” Study...resource selection Methodic review of 483 potential resources to establish the 65 unique resources that would be used in the simulations.](https://reader033.fdocuments.us/reader033/viewer/2022041610/5e37796c21712a142736da2e/html5/thumbnails/11.jpg)
Self-reported instruments
Simulation
Data Cleaning & Quality Control
Codebook Creation
Takeaways
Q & A
@RSICStudy
![Page 12: Designing a “Point of Selection” Study...resource selection Methodic review of 483 potential resources to establish the 65 unique resources that would be used in the simulations.](https://reader033.fdocuments.us/reader033/viewer/2022041610/5e37796c21712a142736da2e/html5/thumbnails/12.jpg)
prescreen survey
● Demographics
● Internet access and use
● Librarian help in the last two years
● Participation in follow-up study
@RSICStudy
![Page 13: Designing a “Point of Selection” Study...resource selection Methodic review of 483 potential resources to establish the 65 unique resources that would be used in the simulations.](https://reader033.fdocuments.us/reader033/viewer/2022041610/5e37796c21712a142736da2e/html5/thumbnails/13.jpg)
29 Grades 4-5
30 Grades 6-8
26 Grades 9-12
30 Community College Students
30 Undergraduate Students
30 Graduate Students
our cohorts
@RSICStudy
![Page 14: Designing a “Point of Selection” Study...resource selection Methodic review of 483 potential resources to establish the 65 unique resources that would be used in the simulations.](https://reader033.fdocuments.us/reader033/viewer/2022041610/5e37796c21712a142736da2e/html5/thumbnails/14.jpg)
interview questionsPre-Simulation● Librarian/library interaction ● # of research projects in the last two years● Their last research project ● Device use● Social media use for school
Post-Simulation● Confidence● Care● Container @RSICStudy
![Page 15: Designing a “Point of Selection” Study...resource selection Methodic review of 483 potential resources to establish the 65 unique resources that would be used in the simulations.](https://reader033.fdocuments.us/reader033/viewer/2022041610/5e37796c21712a142736da2e/html5/thumbnails/15.jpg)
Self-reported instruments
Simulation
Data Cleaning & Quality Control
Codebook Creation
Takeaways
Q & A
@RSICStudy
![Page 16: Designing a “Point of Selection” Study...resource selection Methodic review of 483 potential resources to establish the 65 unique resources that would be used in the simulations.](https://reader033.fdocuments.us/reader033/viewer/2022041610/5e37796c21712a142736da2e/html5/thumbnails/16.jpg)
What’s a simulation?
https://commons.wikimedia.org/wiki/File:Rollover_simulator_J1.jpg@RSICStudy
![Page 17: Designing a “Point of Selection” Study...resource selection Methodic review of 483 potential resources to establish the 65 unique resources that would be used in the simulations.](https://reader033.fdocuments.us/reader033/viewer/2022041610/5e37796c21712a142736da2e/html5/thumbnails/17.jpg)
Why a simulation?
https://commons.wikimedia.org/wiki/File%3ANeutral_Buoyancy_Simulator_cutaway.jpg@RSICStudy
![Page 18: Designing a “Point of Selection” Study...resource selection Methodic review of 483 potential resources to establish the 65 unique resources that would be used in the simulations.](https://reader033.fdocuments.us/reader033/viewer/2022041610/5e37796c21712a142736da2e/html5/thumbnails/18.jpg)
@RSICStudy
![Page 19: Designing a “Point of Selection” Study...resource selection Methodic review of 483 potential resources to establish the 65 unique resources that would be used in the simulations.](https://reader033.fdocuments.us/reader033/viewer/2022041610/5e37796c21712a142736da2e/html5/thumbnails/19.jpg)
Partnerships: advisory panelRepresentation:
● Public Librarian● School Librarian● Academic (Community College) ● Librarian Grade 4-5 ● Instructor Grade 6-8 ● Instructor Grade 9-12 ● College Instructor● University Instructor
@RSICStudy
![Page 20: Designing a “Point of Selection” Study...resource selection Methodic review of 483 potential resources to establish the 65 unique resources that would be used in the simulations.](https://reader033.fdocuments.us/reader033/viewer/2022041610/5e37796c21712a142736da2e/html5/thumbnails/20.jpg)
advisory panel activities: research prompts and resource lists
One initial face-to-face meeting at UF
Formation of six small teams based on the cohorts of students used in study.
Several Google Hangouts working group meetings
Led by a member of the research team, small teams met virtually to:
● Establish a grade-level, standards-aligned research prompt for the simulation ● Determine appropriate number of resources to present to present in simulated results lists and
number of resources participants would be asked to select from the results (cognitive load)● Curate a list of appropriate resources for potential inclusion in each simulation for their
educational levels@RSICStudy
![Page 21: Designing a “Point of Selection” Study...resource selection Methodic review of 483 potential resources to establish the 65 unique resources that would be used in the simulations.](https://reader033.fdocuments.us/reader033/viewer/2022041610/5e37796c21712a142736da2e/html5/thumbnails/21.jpg)
why a python?
@RSICStudy
![Page 22: Designing a “Point of Selection” Study...resource selection Methodic review of 483 potential resources to establish the 65 unique resources that would be used in the simulations.](https://reader033.fdocuments.us/reader033/viewer/2022041610/5e37796c21712a142736da2e/html5/thumbnails/22.jpg)
single, controversial science topic
regional interest
serious gore appeal!
understood by local participants
realistically studied
why a python?CHECKLIST
![Page 23: Designing a “Point of Selection” Study...resource selection Methodic review of 483 potential resources to establish the 65 unique resources that would be used in the simulations.](https://reader033.fdocuments.us/reader033/viewer/2022041610/5e37796c21712a142736da2e/html5/thumbnails/23.jpg)
resource selectionMethodic review of 483 potential resources to establish the 65 unique resources that would be used in the simulations.
Small teams searched for & identified potential resources targeting assigned cohort (graduate, undergraduate, community college, Grade 9-12,Grade 6-8, or Grade 4-5).
Google Drive Sheets to curate, access and review suggested titles. Moved to a single spreadsheet because of overlap
Resources reviewed by teams following dimensions of credibility:● Currency/Timeliness● Authority● Objectivity● Accuracy● Good coverage● Relevance.
Final sources selected based on teams’ 1-3 rankings
![Page 24: Designing a “Point of Selection” Study...resource selection Methodic review of 483 potential resources to establish the 65 unique resources that would be used in the simulations.](https://reader033.fdocuments.us/reader033/viewer/2022041610/5e37796c21712a142736da2e/html5/thumbnails/24.jpg)
research promptsElementary
High School
Middle School
@RSICStudy
![Page 25: Designing a “Point of Selection” Study...resource selection Methodic review of 483 potential resources to establish the 65 unique resources that would be used in the simulations.](https://reader033.fdocuments.us/reader033/viewer/2022041610/5e37796c21712a142736da2e/html5/thumbnails/25.jpg)
research promptsCommunity College
Graduate
Undergraduate
@RSICStudy
![Page 26: Designing a “Point of Selection” Study...resource selection Methodic review of 483 potential resources to establish the 65 unique resources that would be used in the simulations.](https://reader033.fdocuments.us/reader033/viewer/2022041610/5e37796c21712a142736da2e/html5/thumbnails/26.jpg)
https://www.nextgenscience.org @RSICStudy
![Page 27: Designing a “Point of Selection” Study...resource selection Methodic review of 483 potential resources to establish the 65 unique resources that would be used in the simulations.](https://reader033.fdocuments.us/reader033/viewer/2022041610/5e37796c21712a142736da2e/html5/thumbnails/27.jpg)
seven common thread resources
@RSICStudy
BK06: The Effects of Burmese Pythons on Everglades Ecosystemshttp://www.csus.edu/envs/Documents/Theses/Spring%202015/851.2015.Spring.pdf
BK07: Invasive Pythons in the United States: Ecology of an Introduced Predatorhttps://books.google.com/books?id=6KkFcpAHGl8C&printsec=frontcover&source=gbs_ge_summary_r&cad=0#v=onepage&q&f=false
BL04: Everglades Python Challenge: The Hunt for an Invasive Specieshttp://www.captainmitchs.com/airboat,tour,ride,evergladescity,captainmitch,adventures,capt,fishing,fish,/everglades-burmese-python/
JL06: Ecology: Snakes wipe out Everglades rabbits.http://www.nature.com/nature/journal/v519/n7544/full/519393d.html
WE01: Everglades Under Attack. A Burmese Python Invasion.http://pythonpatrol.weebly.com/
WE02: Burmese Pythons: Researchhttp://www.nps.gov/ever/learn/nature/burmesepythonresearch.htm
WE06: Burmese pythons in Floridahttps://en.wikipedia.org/wiki/Burmese_pythons_in_Florida
![Page 28: Designing a “Point of Selection” Study...resource selection Methodic review of 483 potential resources to establish the 65 unique resources that would be used in the simulations.](https://reader033.fdocuments.us/reader033/viewer/2022041610/5e37796c21712a142736da2e/html5/thumbnails/28.jpg)
partnerships: instructional designer & programmer
Build Software:
Articulate’s Storyline 2
Data Extraction Software:
Uniform Server
@RSICStudy
![Page 29: Designing a “Point of Selection” Study...resource selection Methodic review of 483 potential resources to establish the 65 unique resources that would be used in the simulations.](https://reader033.fdocuments.us/reader033/viewer/2022041610/5e37796c21712a142736da2e/html5/thumbnails/29.jpg)
JL06CITESCResource Task Click
@RSICStudy
![Page 30: Designing a “Point of Selection” Study...resource selection Methodic review of 483 potential resources to establish the 65 unique resources that would be used in the simulations.](https://reader033.fdocuments.us/reader033/viewer/2022041610/5e37796c21712a142736da2e/html5/thumbnails/30.jpg)
@RSICStudy
![Page 31: Designing a “Point of Selection” Study...resource selection Methodic review of 483 potential resources to establish the 65 unique resources that would be used in the simulations.](https://reader033.fdocuments.us/reader033/viewer/2022041610/5e37796c21712a142736da2e/html5/thumbnails/31.jpg)
Self-reported instruments
Simulation
Data Cleaning & Quality Control
Codebook Creation
Takeaways
Q & A
@RSICStudy
![Page 32: Designing a “Point of Selection” Study...resource selection Methodic review of 483 potential resources to establish the 65 unique resources that would be used in the simulations.](https://reader033.fdocuments.us/reader033/viewer/2022041610/5e37796c21712a142736da2e/html5/thumbnails/32.jpg)
quantitative data cleaning & preparationCombining Raw Data
1. Combine survey and interview responses for each cohort
2. Combine all cohorts’ survey and interview data with all cohorts storyline data
Cleaning DataCreating Variables1. Facilitate combining cohorts
2. Quantify open-ended text responses
3. Replace facilitator text input with more accurate transcription of participants’ audio recorded responses
4. Capture additional information not included in raw data
1. Correct inaccurate and incomplete cases
2. Change cases to indicate that no response was given when appropriate
3. Remove redundancies
@RSICStudy
![Page 33: Designing a “Point of Selection” Study...resource selection Methodic review of 483 potential resources to establish the 65 unique resources that would be used in the simulations.](https://reader033.fdocuments.us/reader033/viewer/2022041610/5e37796c21712a142736da2e/html5/thumbnails/33.jpg)
qualitative data cleaning and preparation
1. Select a transcription service
2. Send audio for transcription
3. Review, clean, and scrub transcripts
@RSICStudy
![Page 34: Designing a “Point of Selection” Study...resource selection Methodic review of 483 potential resources to establish the 65 unique resources that would be used in the simulations.](https://reader033.fdocuments.us/reader033/viewer/2022041610/5e37796c21712a142736da2e/html5/thumbnails/34.jpg)
Self-reported instruments
Simulation
Data Cleaning & Quality Control
Codebook Creation
Takeaways
Q & A
@RSICStudy
![Page 35: Designing a “Point of Selection” Study...resource selection Methodic review of 483 potential resources to establish the 65 unique resources that would be used in the simulations.](https://reader033.fdocuments.us/reader033/viewer/2022041610/5e37796c21712a142736da2e/html5/thumbnails/35.jpg)
what is coding?
“The goal of coding is ultimately data reduction…”(Connaway & Radford, 2017, p. 299)
@RSICStudy
![Page 36: Designing a “Point of Selection” Study...resource selection Methodic review of 483 potential resources to establish the 65 unique resources that would be used in the simulations.](https://reader033.fdocuments.us/reader033/viewer/2022041610/5e37796c21712a142736da2e/html5/thumbnails/36.jpg)
Initial Meeting
Clean-up
Trial Coding Edit and Add Examples
Review K12 Transcripts
Meeting to Approve New
Codes
Add Definitions and Examples
FtF Meeting
ICR
@RSICStudy
![Page 37: Designing a “Point of Selection” Study...resource selection Methodic review of 483 potential resources to establish the 65 unique resources that would be used in the simulations.](https://reader033.fdocuments.us/reader033/viewer/2022041610/5e37796c21712a142736da2e/html5/thumbnails/37.jpg)
• 3 transcripts from each cohort
• Variety of session lengths
• Recommendations from facilitators
selecting transcripts
@RSICStudy
![Page 38: Designing a “Point of Selection” Study...resource selection Methodic review of 483 potential resources to establish the 65 unique resources that would be used in the simulations.](https://reader033.fdocuments.us/reader033/viewer/2022041610/5e37796c21712a142736da2e/html5/thumbnails/38.jpg)
sources for themes and codes
• Transcript analysis
• Literature
• Data collection instrumentsRecip
e
@RSICStudy
![Page 39: Designing a “Point of Selection” Study...resource selection Methodic review of 483 potential resources to establish the 65 unique resources that would be used in the simulations.](https://reader033.fdocuments.us/reader033/viewer/2022041610/5e37796c21712a142736da2e/html5/thumbnails/39.jpg)
codebook structure• Based on:
• Patterns in identified themes
• Research questions
• 3 Sections
• Cues
• Judgments
• Personal Commentary@RSICStudy
![Page 40: Designing a “Point of Selection” Study...resource selection Methodic review of 483 potential resources to establish the 65 unique resources that would be used in the simulations.](https://reader033.fdocuments.us/reader033/viewer/2022041610/5e37796c21712a142736da2e/html5/thumbnails/40.jpg)
fun with coding
![Page 41: Designing a “Point of Selection” Study...resource selection Methodic review of 483 potential resources to establish the 65 unique resources that would be used in the simulations.](https://reader033.fdocuments.us/reader033/viewer/2022041610/5e37796c21712a142736da2e/html5/thumbnails/41.jpg)
are these the same concept?• “This is kind of like a counterargument of what I would say. So it's
going to have both sides.” (U23 45:04)
• “So this is interesting because the title is actually suggesting that it is going to take a different viewpoint” (H20 25:48)
• “I don't like Wikipedia because other people can give you different information and their opinion, so I don't use Wikipedia.” (E14 04:07)
• “TIME Magazine I'll just put somewhat credible because I'm sure there's some kind of bias involved in the magazine setting.” (G01 48:16) @RSICStudy
![Page 42: Designing a “Point of Selection” Study...resource selection Methodic review of 483 potential resources to establish the 65 unique resources that would be used in the simulations.](https://reader033.fdocuments.us/reader033/viewer/2022041610/5e37796c21712a142736da2e/html5/thumbnails/42.jpg)
![Page 43: Designing a “Point of Selection” Study...resource selection Methodic review of 483 potential resources to establish the 65 unique resources that would be used in the simulations.](https://reader033.fdocuments.us/reader033/viewer/2022041610/5e37796c21712a142736da2e/html5/thumbnails/43.jpg)
inter-coder reliability
![Page 44: Designing a “Point of Selection” Study...resource selection Methodic review of 483 potential resources to establish the 65 unique resources that would be used in the simulations.](https://reader033.fdocuments.us/reader033/viewer/2022041610/5e37796c21712a142736da2e/html5/thumbnails/44.jpg)
Self-reported instruments
Simulation
Data Cleaning & Quality Control
Codebook Creation
Takeaways
Q & A
@RSICStudy
![Page 45: Designing a “Point of Selection” Study...resource selection Methodic review of 483 potential resources to establish the 65 unique resources that would be used in the simulations.](https://reader033.fdocuments.us/reader033/viewer/2022041610/5e37796c21712a142736da2e/html5/thumbnails/45.jpg)
![Page 46: Designing a “Point of Selection” Study...resource selection Methodic review of 483 potential resources to establish the 65 unique resources that would be used in the simulations.](https://reader033.fdocuments.us/reader033/viewer/2022041610/5e37796c21712a142736da2e/html5/thumbnails/46.jpg)
![Page 47: Designing a “Point of Selection” Study...resource selection Methodic review of 483 potential resources to establish the 65 unique resources that would be used in the simulations.](https://reader033.fdocuments.us/reader033/viewer/2022041610/5e37796c21712a142736da2e/html5/thumbnails/47.jpg)
![Page 48: Designing a “Point of Selection” Study...resource selection Methodic review of 483 potential resources to establish the 65 unique resources that would be used in the simulations.](https://reader033.fdocuments.us/reader033/viewer/2022041610/5e37796c21712a142736da2e/html5/thumbnails/48.jpg)
![Page 49: Designing a “Point of Selection” Study...resource selection Methodic review of 483 potential resources to establish the 65 unique resources that would be used in the simulations.](https://reader033.fdocuments.us/reader033/viewer/2022041610/5e37796c21712a142736da2e/html5/thumbnails/49.jpg)
![Page 50: Designing a “Point of Selection” Study...resource selection Methodic review of 483 potential resources to establish the 65 unique resources that would be used in the simulations.](https://reader033.fdocuments.us/reader033/viewer/2022041610/5e37796c21712a142736da2e/html5/thumbnails/50.jpg)
![Page 51: Designing a “Point of Selection” Study...resource selection Methodic review of 483 potential resources to establish the 65 unique resources that would be used in the simulations.](https://reader033.fdocuments.us/reader033/viewer/2022041610/5e37796c21712a142736da2e/html5/thumbnails/51.jpg)
![Page 52: Designing a “Point of Selection” Study...resource selection Methodic review of 483 potential resources to establish the 65 unique resources that would be used in the simulations.](https://reader033.fdocuments.us/reader033/viewer/2022041610/5e37796c21712a142736da2e/html5/thumbnails/52.jpg)
Self-reported instruments
Simulation
Data Cleaning & Quality Control
Codebook Creation
Takeaways
Q & A
@RSICStudy
![Page 53: Designing a “Point of Selection” Study...resource selection Methodic review of 483 potential resources to establish the 65 unique resources that would be used in the simulations.](https://reader033.fdocuments.us/reader033/viewer/2022041610/5e37796c21712a142736da2e/html5/thumbnails/53.jpg)
We Made It!
Questions?
@RSICStudy