1 Michigan Teacher Network michiganteacher.net Transaction Log Analysis & Michigan Teacher Network...
-
Upload
camilla-goodman -
Category
Documents
-
view
218 -
download
0
Transcript of 1 Michigan Teacher Network michiganteacher.net Transaction Log Analysis & Michigan Teacher Network...
1
Michigan Teacher Network • michiganteacher.net
Transaction Log Analysis & Michigan Teacher Network
Marcia A. MardisMerit Network at the University of
Michigan
2
Michigan Teacher Network • michiganteacher.net
Working Definitions User centered design is a methodology that concentrates on product
design and development from user, user task, and user interface perspectives. A UCD process functions in parallel with a product development process to:
Collect Data: About users, their tasks, and their current work processes. Analyze Data: To determine how to redesign the workflow, and to identify usability
requirements and goals. Design and Develop: Scenarios, information architectures and grammars, and user
interface prototypes. Test: Design and conduct iterations of usability assessments and tests. Deploy: Follow up with measures of usability and user satisfaction
Transaction log analysis (TLA) uses transaction logs to discern attributes of the search process, such as the searcher’s actions, the interaction between the users and the system, and the evaluation of the results by the searcher. (Jansen & Pooch, 2000).
3
Michigan Teacher Network • michiganteacher.net
What can you do with TLA?
Characterize user sessions Length of time and time of day Number of pages viewed Where coming from
Characterize of user queries Number of revisions Number of terms used Use of logic and advanced search techniques
Characterize search terms Topic Frequency of terms
4
Michigan Teacher Network • michiganteacher.net
Key Authors & Studies
Spink, Jansen, Wolfram & Saracevic Excite studies Meta-evaluation Frameworks, general taxonomies
Bryan-Kinns & Blandford Digital library TLA (vs. Web TLA) Interactional traps (misunderstanding; premature bailout)
Silverstein & Henziger AltaVista search logs Comparison of theoretical vs. actual user models
Abbas Teachers & Children’s info seeking (her own work and lit review) Influence of metadata and search scaffolds
5
Michigan Teacher Network • michiganteacher.net
Consistent Findings from Studies
Very few queries use advanced search features like boolean; half of those that do contain mistakes
Highly skewed use of terms: few used a lot, a lot used infrequently
Distribution of terms bears no relation to distribution of content
Users look mainly at the first page of results only
Most sessions contain only 1 query Most queries are very short (3 terms or less)
6
Michigan Teacher Network • michiganteacher.net
Caveats
Are the exhibited searching characteristics due to some unique aspect of the system or collection?
Are the user samples from the same or different populations?
Are the searching characteristics really applicable to all users?
What are the implications of users who do not understand what they are searching (metadata vs. full-text)
What is the tradeoff between user privacy and data collection?
Real searchers have different information seeking processes Are the topic classification schemes used consistently? Is the data collection tool accurate?
7
Michigan Teacher Network • michiganteacher.net
What are information seeking
behaviors? Actions and expressions exhibited
(recursively) during: Query formation
Topic and concept identification Keyword generation
Information collection Source selection Source navigation (Searching/browsing)
Results evaluation Quality reflection Query revision Source reselection (choose another tool)
8
Michigan Teacher Network • michiganteacher.net
Why is it important to look at these
behaviors? For tool developers
User interface design Training/Help content and design Knowledge of where need to ask questions might arise
For questioners Understand where asking questions may be integrated into
information seeking process
For responders Phrasing and types of answers given Understanding of barriers to use
9
Michigan Teacher Network • michiganteacher.net
What’s consistent in the literature?
Time is consistently a big factor in teacher information seeking (Fitzgerald, 2001; Pittsley, 2000, etc.)
Teachers say quality is important (Sutton, 1999; Fitzgerald, 2001; Mardis, 2003, etc.)
Teachers don’t take time to evaluate quality (Lazonder, 2001)
Teachers are willing to interact (Fitzgerald, 2001; Shumar, 2003; Renninger, 2003)
Teachers consult colleagues (Sutton, 1999; Mardis & Hoffman, 2003, etc.)
10
Michigan Teacher Network • michiganteacher.net
Higher ed faculty and K-12 teacher
behaviors: compared Instructors Primary research (use of email, discussion groups) fieldwork
data, archival materials, and gray literature Citation rhetoric (Bates, 2001) Very little instruction on using Web for course resources
Teachers Information tends to be for self-generated queries Teachers adapt resources (Sutton, 1999; Fitzgerald, 2001) Teachers use each other as primary sources of information
11
Michigan Teacher Network • michiganteacher.net
Adults and K-12 teacher behaviors:
compared Topic familiarity determines amount of time
spent searching and reading. (Kelly & Cool, 2002)
Both groups are not familiar with search engine structure and function (Bradley, 1999)
No structured search models exist for either group
12
Michigan Teacher Network • michiganteacher.net
Children and K-12 teacher behaviors:
compared Little extensive, controlled study has been
done (Dresang, 1999) Children overperceive their abilities to use
search tools; teachers bail out more quickly (Dresang, 1999; Fidel, 1999)
Both groups excel in systems designed for them (Hirsh, 1997; Dresang, 1999)
Younger students and teachers favor browsing (Dresang, 1999)
13
Michigan Teacher Network • michiganteacher.net
What are some key ideas?
Save time Describe richly (Sutton, 1999) Quick response is essential Recommend ready-to-use resources with clear application and
evaluation/assessment Assure quality (rating/peer review) (anecdotal) Question formation is still weak link Don’t assume technology skills can/will be
gained(Weglinksy, 2000) Users need a lot of support (documentation, online help)
(Pollock, 1999)
14
Michigan Teacher Network • michiganteacher.net
About Michigan Teacher Network
(MTN) Founded in 1998 as state clearinghouse Contains over 8500 professional learning,
professional support, and curriculum resources correlated to the Michigan Curriculum Framework
Receives additional funding from the NSF NSDL Receives 500,000 (summer)-700,000(fall/winter)
page views per day Executes 750 (summer)-950(fall/winter) search
strings per day Governed by Merit privacy policy
15
Michigan Teacher Network • michiganteacher.net
Research Questions
What do MTN users look for? Do MTN users look for classroom planning
information at home? Do users use too many or too few search
terms? Do users seem to understand what they’re
searching?
16
Michigan Teacher Network • michiganteacher.net
Characteristics of Windows Examined
2/19-3/5 3/26-4/9 4/29-5/14
N 10870 7462 5896
Mean Searches per day 776 530 420
Mean words per string 2.48 1.97 1.99
Mean time of search 3:12pm 2:40pm 2:38pm
17
Michigan Teacher Network • michiganteacher.net
Broad Category of Searches
Search Category
3000
2000
1000
0
Data collected from April 29-May 14, 2003
18
Michigan Teacher Network • michiganteacher.net
Top Searches 2/19-3/5/2003
classroom management 300
sciences 102
lesson plans 90
sound 87
states 91
history 72
social studies 71
economics 64
elementary education 64
technology education 58
19
Michigan Teacher Network • michiganteacher.net
Top Searches 3/26-4/9/2003
classroom management 304
physical education 60
lesson plans 55
educational assessment 51
fractions 47
private schools 41
weather 39
rainforests 35
music 34
life science 31
20
Michigan Teacher Network • michiganteacher.net
Top Searches 4/29- 5/14/2003
classroom management 180
listserv(s) 146
poetry of kids 109
science 48
lesson plans 41
private schools 37
zookeeper 31
plants 30
plagiarism 28
organizational skills 27
21
Michigan Teacher Network • michiganteacher.net
Data collected from April 29-May 14, 2003
Curriculum Area Searches
Curriculum Area
800
600
400
200
0
23
Michigan Teacher Network • michiganteacher.net
Conclusions
Many users exhibit “Search Engine Habits” Search for items beyond scope of collection Seem to be expecting to search site text, not metadata Perform searches on browsing categories
24
Michigan Teacher Network • michiganteacher.net
Conclusions
Most searches occur after school MTN users seek classroom management help
the most Science is the most searched curriculum area Users search very broad or very narrow
concepts with few terms Users seem to want objects Users repeatedly type in the same search