1. RemBench: A Digital Workbench for Rembrandt Research Suzan Verberne, Centre for Language and Speech Technology Rudie van Leeuwen, Department of Art History Radboud University Nijmegen Digital Humanities Benelux, June 12, 2014

2. About RemBench RemBench: a Digital Workbench for Rembrandt Research Project financed by CLARIN-NL Goal: show the value of online digital data for art history research - Subgoal 1: connect the metadata of four different databases - Subgoal 2: build a demonstrator search interface to these databases Target group: historians, art historians and other humanities scholars and students Suzan Verberne, May 2014 3. Suzan Verberne, May 2014 4. Suzan Verberne, May 2014 5. About RemBench RemBench connects four existing databases, all accessible through APIs: RKDartists and RKDimages: http://explore.rkd.nl/ RemDoc, a collection of original documents related to Rembrandt (1475-1750): http://www.remdoc.org/ RUQuest, a library system that provides access to the collection of the Raboud University Library, and full-text articles: http://www.ru.nl/library/vm/ruquest/ Suzan Verberne, May 2014 6. Challenges 1. Mapping the metadata schemes of the four databases onto one set of metadata categories; 2. Developing the technology that allows the user to search all databases at once; 3. Designing a user interface that is intuitive for our target group. Suzan Verberne, May 2014 7. Evaluation: user study Research questions: 1. How satisfied are target group users with RemBench? 2. Which features are used by the participants and which are not? 3. How efficient are the users in finding answers to Rembrandt research questions using RemBench? Suzan Verberne, May 2014 8. User study Design: Participants: 9 students (art history and history, bachelor and master) Task: try to find the best possible answer(s) to max. 10 questions related to Rembrandt, using RemBench Observation of user-system interaction - (Guided) thinking-aloud - Desktop activity recording Post-task questionnaire Suzan Verberne, May 2014 9. User study 61 questions formulated by a Rembrandt expert, e.g. - Wanneer is de Staalmeesters geschilderd? - Waar heeft de Nachtwacht (in de loop der tijd) gehangen (herkomstgeschiedenis)? - Welke werken van Rembrandt bevinden zich in St. Petersburg? - Hoe vaak schilderde Rembrandt Christus? - Is Rembrandts Lezende Vrouw in het Rijksmuseum geschilderd op doek of paneel? - Had Rembrandt kleinkinderen; zo ja, waar zijn die geboren? Questionnaire per question: - What was the result of your search? - How satisfied are you with the answer(s) found? (5-point Likert scale) - How satisfied are you with the use of RemBench for answering the question? (5-point Likert scale) Suzan Verberne, May 2014 10. User study The participant is asked to use RemBench exhaustively to find the answer(s) When he/she feels that he tried everything to find the answer(s), he/she stops his search After all questions, he/she is asked to fill in the post-task questionnaire - Please list the positive aspects of RemBench - Please list the negative aspects of RemBench Suzan Verberne, May 2014 11. Example of user interaction Suzan Verberne, May 2014 12. Quantitative evaluation # of questions 61 # of questions with at least one user evaluation 54 # of questions with two user evaluations 15 Suzan Verberne, May 2014 Mean difference between the two users' evaluations 1.3 1.0 13. Qualitative evaluation All functionalities are used, except for Refresh (clear all), boolean query options and fuzzy search. Positive aspects: Clear colours, good font type, nice overview, good layout, easy to use Search options are immediately clear Searching possible in English and Dutch Good filter options (esp. date range and location) Works of art, primary and secondary sources combined Results ordered by type Lots of information Reliable sources in the search results References to RKD explore with even more information Suzan Verberne, May 2014 14. Qualitative evaluation Negative aspects: Options missing: - Specific topics of works of art - Search for works by one specific artist (Rembrandt) - Person search should not only cover artists but also family members etc. Difficult to find specific answers Differences in results between English and Dutch queries Sometimes difficult to choose between many search options Difficult to get the expected number of results: either too many or too few Searching sometimes takes a lot of time No explanation on the web site Low quality of images Suzan Verberne, May 2014 15. Recommendations for the final RemBench demonstrator Suzan Verberne, May 2014 Add a facet 'Artist/author name' with search field, to make it possible to find works of 1 specific artist. Change the name of the facet 'Person type' to 'Artist/author type and change the caption Persons in the result list to Artists Make it possible to enter specific years in the date filter Solve the problem of too many irrelevant results for queries containing the term Rembrandt Make the use of the refresh button more clear How to add explanation of use of boolean AND, phrasal queries, and fuzzy search? 16. Recommendations for usability studies In our case, 9 subjects was enough to reach saturation of evaluation points Guided thinking-aloud protocol successful. Make extensive notes, no microphone recordings necessary. Screen capture more useful than camera recordings (e.g. with Blueberry Flashback Express) Dont make the tasks too difficult. Some of our questions where too complex (esp. list all..., how many..., requiring exhaustive search) Explain elaborately that you are testing the web site, not the subject. Suzan Verberne, May 2014 17. Try RemBench yourself Come to our demo presentation this afternoon Suzan Verberne, May 2014