1. Sanjeet MannMeasuring ElectronicUniversity of RedlandsSCELC Research DayResource Availability March 5, 2013

2. Armacost Library infrastructureILLIAD interlibrary loansystem Full text targets (databases, ejournals) (79,757 unique titles) Serials Solutions 360 Link A&I databases (~77,000 titles indexed) Innovative Interfaces catalog/proxy 3. Why do electronic resource errors matter? Costs Frustrated expectations Undermined confidence Complicated instruction 4. Research question "How often doesfull text linking work?" 5. Availabilitystudies Sample of items Available? Yes/No Error? Order encountered Probabilities Prioritize fixes 6. Development of the availability techniquePrint material availabilitycard catalog user surveys(Reviewed in Mansbridge 1986, Nisonger 2007)Linear sequence(De Prospo 1973)Branching model(Kantor 1976)Applied to e-resources500 articles from 50 high impact journals(Nisonger 2009) 7. OpenURL performanceOpenURL-based reasonsfor availability error(Wakimoto et al. 1998)Digging into the Data onlink resolver failure(Trainor and Price 2010)NISO Initiatives:KBART, IOTA, PIE-J(Chandler et al. 2011, Glasser 2012, Kasprowski 2012) 8. Usability studies focusing on e-resourcesDatabase linkpages(Fry 2011, Ponsford et al. 2011b)Resolver menus(ONeill 2009, Imler & Eichelberger 2011, Ponsford et al. 2011a)Discoveryservices(Williams & Foster 2011, Fagan et al. 2012)Entire process 9. Methodology400 citations 4 questionsX 10 databasesX 10 results[18:11] redlandsreference:what is your research topic? Arts & Humanities[18:11] meeboguest59808: RILMOral Motor ActivityMLA Philosophers Index[18:11] redlandsreference:Is this for a CommunicativeDisorders class? Social Sciences America: History & Life EconLit Sociological Index Sciences Biological Abstracts ComDisDome, 10. Link testing http://goo.gl/606us 11. Error coding What is an error? Six errorcategories Updated criteria 12. Armacost Library failure points: P vs. O 13. Error details 1: Proxy errorsDomain missing fromforward tableDomain missing from SSLCertificateTimeouts trying toestablish connection 14. Error details 2: Source errors Missing metadata Erroneous metadata (e.g. rft.date=0001-01-01) 15. Error details 3: Knowledge base errorsTitle not selected inknowledge baseTitle selected, but inpoorly chosen collectionKnowledge baseholdings do not reflectaccess entitlement(embargo, back issues,etc.) 16. Error details 4: Link resolver errorConfusion between twosimilar titlesUnusual OpenURL syntax 17. Error details 5: Target errors Content not loaded (supplement, embargo) Records concatenated from full text and non-full- text databases Server downtime 18. Error details 6: ILLIAD errors Unicode metadata not displayed properly rft.title used for both book title and article title, affects chapters and dissertations 19. Results Error rate: 150/400 (38%) Overall availability: 250/400 (62%) Locally downloadable: 104/400 (26%) 20. SamplingNecessary sample size for a yes/no condition is determined by:To use this, you need:Availability rate from a small pre-testChoose acceptable % confidence (95%)Choose acceptable margin of error (+/- 5%)Plug values into the formulap = 0.625 (250 / 400 successes)1-p = 0.375 (150 / 400 errors)C = 0.95 (95% confidence)Zc = 1.96 (statistical textbook orhttp://www.measuringusability.com/pcalcz.php)E = 0.05 (5% error)I could have just used 360 citations 21. ConfidenceYour confidence in a study of a particular sample size is given by:I could have just used 360 citations 22. Discussion 23. Discussion (continued) 24. Local Availability / Errors by Discipline 25. SolutionsEdit proxy forwardtable 26. SolutionsUpgradeILLIAD 27. SolutionsCustomizeSerialsSolutions 28. SolutionsSimplify result screen interface and terminology 29. Summary400 citations obtained through likely keyword searchesof 10 A&I databases62% availability / 38% error rate (98% confidence, +/- 5%)26% downloadable full textResponses include fixing proxy, kb holdings, interfaces,upgrading systemsStrengths: quant + qual data, very flexible(n=100 allows 85% confidence)Weaknesses: Does not account for issues withinterfaces, searching or evaluation faced by actual users 30. Towards availability testing with live students More barriers:o confusing interfaceso difficulty formulatingsearches and evaluatingsourceso login errors How to test:o cognitive walkthrough +recorded task protocolso analysis informsinformation literacy andinterface design Deliverables:o availability %o branching modelo usability report 31. QuestionsFurther reading: http://goo.gl/4fu47Data set: http://goo.gl/606us skatdesign.co m