Leveraging Interfaces to Improve Recommendation DiversityChun-Hua Tsai

University of Pi�sburgh135 N Belle�eld AvePi�sburgh, PA 15260

cht77@pi�.edu

Peter BrusilovskyUniversity of Pi�sburgh

135 N Belle�eld AvePi�sburgh, PA 15260

peterb@pi�.edu

ABSTRACTIncreasing diversity in the output of a recommender system is anactive research question for solving a long-tail issue. Most of thecurrent approaches have focused on ranked list optimization toimprove recommendation diversity. However, li�le is known aboutthe e�ect that a visual interface can have on this issue. �is papershows that a multidimensional visualization promotes diversityof social exploration in the context of an academic conference.Our study shows a signi�cant di�erence in the exploration pa�ernbetween ranked list and visual interfaces. �e results show that avisual interface can help the user explore a a more diverse set ofrecommended items.

KEYWORDSHCI; Recommender System; Diversity; User Interface

1 INTRODUCTIONA recommender system is an e�ective approach to prevent cognitiveoverload when accessing information. In an era of data, usersare more reliant on such systems to access on-line information.�ese systems can o�er personalized recommendations that arebased on users’ preference or interests. However, personalizedalgorithms tend to push the user to select from a narrow set ofchoices. �is can lead to a �lter bubble e�ect, which shields theuser from other viewpoints [5]. It can also cause an adverse e�ectin social fragmentation and ideological polarization of discussionson social issues [14]. Many studies have been done to prevent thesenegative e�ects by generating diversity in di�erent disciplines, suchas in online reviews [7, 30], comments [8], e-commerce [10, 23],question-answering sites [22], and politics [5, 9].

�ere are two main directions of work toward improving thediversity of recommendation results. �e �rst direction, where alarge portion of e�ort has been spent, focuses on improving therecommendation diversity of a ranked list of results. �is has mainlyfocused on optimization by adding the “beyond relevance” factorsto relevance prediction functions [2, 11, 22, 27, 32, 35]; that is, toprevent issues of over-speci�cation by providing a recommendationlist that contains diverse items. �e second direction is to improvethe interface of the recommender system in order to solve the

Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor pro�t or commercial advantage and that copies bear this notice and the full citationon the �rst page. Copyrights for components of this work owned by others than ACMmust be honored. Abstracting with credit is permi�ed. To copy otherwise, or republish,to post on servers or to redistribute to lists, requires prior speci�c permission and/or afee. Request permissions from permissions@acm.org.UMAP’17 Adjunct,, July 09-12, 2017, Bratislava, Slovakia© 2017 ACM. 978-1-4503-5067-9/17/07. . .$15.00DOI: 10.1145/3099023.3099073

problem. For example, [7, 10, 23, 30] adopted the visual interfaceto present recommendation results with diversity aspects. �euser can interact with the system beyond the ranked list. �einterface represents the recommendation results with a variety ofsimilar/dissimilar items, which can help a user to explore diverserecommendation content.

It is not surprising that all the methods for the presentation ofrecommendations include some kind of bias. For example, manystudies pointed out the display bias in the sorted recommendationlist; that is, the top-ranked items would get more user a�entionthan lower-ranked items [4, 13], or the so-called position bias. Moretypes of bias can be found in a mix or visual design recommendersystems [28]. �e user may have a vertical bias on a ranked list thatis combined with image and search results. �is �nding indicatesthat the interface design does a�ect how the user interacts withthe recommended items. We can say that the ranked list presen-tation aggravates the long-tail e�ect on the recommender system.However, li�le research e�ort has been focused on how the visuale�ect has an impact on the user interaction, as well as the distinctchallenges and opportunities for designing a visual recommendersystem.

In this paper, we propose a user study on comparing a sortedranked list and a two-dimensional visual interface for a social rec-ommendation task at academic conferences. �e experiment resultindicates a signi�cant di�erence in the exploration pa�erns of theseinterfaces. We found that the user visual interface group exploredmore diverse social connections from the system. In the subjec-tive evaluation, however, the ranked list was evaluated higher inuser’s intention to reuse the system. �e main contribution of thispaper is to suggest and evaluate an interface to solve the “long-tail” issue within the recommender system. �e �ndings supportthe overall concept of interface design helping the user to fairlyexplore the recommender system results. We argue that a usercan intuitively access diverse recommendations if we provide thatuser with an accurate perception of how the data is presented.�is �nding provides some insights towards the development of adiversity-supporting interface to access recommendations online.

2 RELATEDWORK2.1 Recommendation Diversity�e recommender system aims to facilitate the presentation ofuseful information and reduce information overload. As a result,an e�cient system is required to deliver the relevant items witha high degree of accuracy, based on user preferences or behavior[24]. Many scholars have criticized highly accurate recommenderalgorithms that provide the user with a narrow set of choices or

Late-Breaking Results, Demonstration and Theory, Opinion & Reflection Paper UMAP’17, July 9-12, 2017, Bratislava, Slovakia

65

limited information exposure [5], which is a well-known over-speci�cation problem in the domain of recommender systems [1,33]. Some scholars argued for the negative �lter bubble e�ect if theuser can only access to a “biased” recommendation result based onpersonalized social network or ideological preferences [9]. It mightbe necessary to diversify the recommendation results to preventthis bias. However, the de�nition of diversity varies and lacks auniversal metric and scale. Diverse items may be generated basedon user personalities for wider exposure [32]. For example, in onlinenews consumption, it may be possible to show di�ering opinionsfor the purposes of comparison [14]. Some studies tried to provide abalanced choice between novel and similar items in a recommenderranked list [11, 27, 29], or to provide categorical coverage that wasas broad as possible [20, 23, 31]. A recommender system that isspeci�cally designed to facilitate the diversity of results needs tocarefully consider the di�erences between the provided results.

2.2 Improving DiversityTo address the diversity problem, much e�ort has been spent onincreasing the diversity of ranked list results o�ered to the usersof recommender systems. �e work can be separated into threecategories: 1) Learning to rank: an approach to aggregate the nov-elty and accuracy in a ranked list. For instance, a hybrid algo-rithm can extend the recommended list with the diversity, novelty,and accuracy of items [2, 11, 15, 22, 27, 32, 35]. 2) Content-baseddiversi�cation: to provide diverse content based on the varyingsimilarity metrics, e.g. opinion similarities [8], aspect similarities[29], and topic similarities [33]. 3) Collaborative �ltering-baseddiversi�cation: to extend the collaborative network by probabilisticapproaches [1, 36] and latent distance [17]. However, all of theseapproaches have focused on the diversi�cation within the rankedlist. �e ensembled ranked list lacked perception of the propertiesbehind the recommendation result. Users o�en lower their satisfac-tion with or trust in the system if it has provided a list that con�ictswith their expected levels of similarity [16].

Providing a visual interface is another approach to solving thediversity recommendation problem. Some examples of a visualdiscovery interface include the CTR rate in an e-commerce web-site [23]; the two-column format to present a two-sided opinionon controversial subjects [14]; and the distance of latency amongusers’ ideology [9] through an explainable interface to justify thereason for providing recommendations for a user to actively ac-cess diverse content [33, 34]. [19] designed an interface to providean explanation to justify the recommendation results, which hasbeen shown to be useful for the user to understand the reasoningbehind particular recommendation results. �e user can chooseto explore the system further, based on the explanation. [21] pro-posed a user-controllable interface that would allow the user tochange each feature�s weighting to provide a more personalizedranking. [7, 10, 30] all proposed various interfaces that show di-verse recommendation results. However, these studies have focusedon individual metrics, such as satisfaction and helpfulness, amongothers. A study from an objective perspective has not yet beenperformed, and it remains unknown as to which interface designcan truly help users consume more diverse results.

Figure 1: A screenshot of the RelExplorer system [26]: (i)A control panel of three feature sliders. (ii) A ranked listof the personalized relevance scores. (iii) �e user pro�leinformation from the Conference Navigator system [6].

2.3 Perceiving DiversityHumans are naturally connected to or prefer to interact with peo-ple who hold similar beliefs - the homophily e�ect [9]. �e socialrecommender system should provide more diverse content for auser to extend their social connections outside of a personal bubble.However, not every user equally values the diversity of providedrecommendations [3]. �e desired level of diversity di�ers fromindividual to individual. For instance, [16] classi�ed people intotwo categories: “diversity seeking” and “challenge averse” to de-scribe the relationship between strati�cation and level of diversityexposure. �is classi�cation may explain the individual di�erencesin the information-seeking process. �e social recommender sys-tem with included diversity needs a di�erent interface to �t theneeds of users’ prior convictions. Furthermore, simply presentingcon�icting information may not help users to interact with diversecontent. A reinforcing e�ect may happen if the user feels threat-ened by unfamiliar information [14]. �e study of [30] adopteddimension-reduction techniques to project the data in two or threedimensions for visualization purposes. However, the user was notable to distinguish the meaning of each axis, which led the user toexplore opinions that were similar to their own [7]. [5] argues thatthe “diverse conceptions of democracy” must be considered whendesigning a diversity-improving tool. It is necessary to realize thevisual e�ect before we can design a general interface for enhancingrecommendation diversity.

3 APPROACH3.1 �e Context�e work presented in this paper has been performed in the contextof developing a social recommendation function for the Confer-ence Navigator system [6]. �is function was added to the originalsystem to help conference a�endees to connect with other schol-ars at the venue. �e overall goal of the social recommendationfunction was to help conference a�endees explore potential sociallinks while a�ending events. Our recommendation approach wasdesigned to emphasize di�erent aspects of similarity that could bevaluable to identify people of interest. �ese aspects were encap-sulated in three separate recommender engines that accounted forthe similarity between publications (academic), co-authorship simi-larities (social), and the similarity of interest in conference papers,expressed as bookmarks (interest). We �rst designed a standard

Late-Breaking Results, Demonstration and Theory, Opinion & Reflection Paper UMAP’17, July 9-12, 2017, Bratislava, Slovakia

66

Figure 2: A two-dimensional scatter plot to combine the rec-ommendation results of two engines. Each axis representsa recommender engine (a feature), each node represents anattendee at the conference, and its location is determined byits featureweights (here - theAcademic feature for Y and theSocial feature for X). A click on a node opens a new windowto explore pro�le details.

recommendation interface with a ranked list and a control panelto change the weighting of contributing engines. �is version wasexplored in the pilot study. �e results of the pilot study brie�y pre-sented in the next subsection helped us to design a more advancedvisual interface, which is presented below.

3.2 �e Pilot StudyIn our pilot study, we explored to what extent users were able toexplore a diverse space of conference a�endees by using a controlledfusion of several focused recommender engines, an approach thatwe successfully used in the past [18]. Figure 1 presents a screenshotof the pilot study interface. �e interface allowed users to fusethree recommendation components into a single. To control thefusion, the user could change the weighting of each componentbased on individual preference. For example, the user may increasethe Academic feature to �nd more conference a�endees with similarresearch interests, or increase the Social feature to �nd conferencea�endees with a similar co-authorship network.

�e results of the pilot study were presented in [26]. In brief,we found that users did use the control panel to explore socialconnections. However, the exploration pa�ern showed that onlya small set of the recommended items had been explored. �at is,most of the exploration focused on the group of people with a highrelevance score. �e group with a low relevance score was lesslikely to be explored. However, the lower relevance score and rankmay be due to a lack of user pro�le data (such as junior scholars whohave fewer publications) or the presentation bias of the ranked listinterface. It does not determine the real social value to users. �epilot study showed the limitations of the ranked-list interface, evenin the presence of user control. �is work motivated the currentstudy presented below: reducing the presentation through a visualinterface.

3.3 Visual InterfaceA ranked list can only represent recommended items in a singledimension. It forces the system to use the ensembled or controlledfusing method to mix multiple aspects of relevance into a singlelist. However, the pilot study showed that the ranked list caused anexploration pa�ern with li�le diversity. To avoid ranking bias, weexplored a two-dimensional interface design that independentlypresents two recommendation aspects. We found that a sca�er plotis a universal design to represent multidimensional data [12]. Hence,we developed a sca�er plot interface to show the recommendedresults in two dimensions (Figure 2).

A sca�er plot diagram allows us to simultaneously present morethan one recommendation feature. For example, we can display theAcademic feature on the X axis and the Social feature on the Y axis.�e two features de�ne the position of each recommended item inthe sca�er plot. �e interface can help users intuitively perceive therelative relationship between the two recommendation features andcan help them quickly focus on a sca�er plot area with a speci�ccombination of two recommendation features (for example, theupper le� corner with high academic and low social similarity), in-stead of changing the feature weighting in the ranked list. To focususer a�ention on areas with meaningful combinations of features,we used color coding to represent four groups of recommendeditems: high relevance in both features (HH), low relevance in bothfeatures (LL), and high relevance in one or the other of the features(HL or LH). We realized the users may �nd it di�cult to understandthe meaning of these four groups, so we set the titles based on theprospective e�ect of each cluster; i.e. the red color of the HH groupindicates the conference a�endees who may be most likely known,due to their high academic and social similarity to the user. �eyellow color of the LL group is the group of a�endees who mayhave a low potential connection, due to the low values of both of thefeatures. �e blue and green colors show the a�endees with onlyone high similarity feature, which are assigned as high potentialconnections and medium potential connections.

Since the features represented in the visualization were com-puted by two di�erent recommender engines, the features were notcomparable without normalization. We adopted a standard Z-Scoreto normalize all the features to the same scale, from 0 to 1. �e func-tion was de�ned as: ZScore = xi−uj

σj , where xi is ith recommendeditem and j represents the corresponding two features. �en, weused a standard Z-table to convert the ZScore to the correspondingpercentile pi j . �is allowed us to list all the features on the samescale for both the ranked list and sca�er plot diagram.

4 EXPERIMENT4.1 Data and ParticipantsWe used the Conference Navigator system with the extended socialrecommendation function [26] for assessing and comparing thetwo proposed interfaces. �e pilot study of the ranked list interfacewas conducted at the HT 2016 and the UMAP 2016 conferences inHalifax, Nova Scotia, Canada. �e study of the visual interface wasperformed at the CIC 2016 conference in Pi�sburgh, Pennsylvania,USA. �e conference publication and a�endee lists were used tocompute recommendations based on three types of similarity (thefeatures mentioned in Section 3.1). For Study 1, we recruited 12

Late-Breaking Results, Demonstration and Theory, Opinion & Reflection Paper UMAP’17, July 9-12, 2017, Bratislava, Slovakia

67

participants (nine male and three female) from the HT&UMAP2016 conferences. For Study 2, we recruited 14 participants (sevenmale and seven female) from the CIC 2016 conference. All of theparticipants were registered conference a�endees with at least onepublication paper in the system. �e age of the participants rangedfrom 20 to 40 years old, and they were recruited at the conference.All of them were graduate students or at the junior research levelat their associated academic institution or industry research lab.

4.2 Experiment Design and ProcedureTo evaluate the e�ect of the proposed interface on the exploration ofsocial recommendations by academic a�endees, we designed a userstudy that combined both a subjective and an objective evaluation.�e ranked list interface (Group 1) was used as the control group,and the sca�er plot interface (Group 2) was the experimental group.We recorded the system activity log while each participant used thesystem. Both studies used the prepared laptop and were performedunder the control of the �rst author directly at the conferencevenue.

We asked participants to complete two tasks using the designatedsystem and to �ll out a post-task questionnaire. �e tasks werebased on di�erent social needs that would arise while using a socialrecommendation system. In each task, participants were asked tofollow two people, according to the instructions. Task1: Find twoconference a�endees you already know. a) Decide whether youneed to follow them or connect to them in the system. b) Examineinformation about these a�endees to �nd out how these two peoplecan help to establish new connections at this conference. Task2:Explore two conference a�endees you a) don�t know in person yetand b) you are interested in meeting. Examine information aboutthese a�endees to �nd out who could introduce you to them orhow you could introduce yourself.

In order to examine to what extent each interface encourageddiversity in exploring conference a�endees, we observed the userinteraction with four di�erent “quadrants” that represented com-binations of two features [25], e.g. high academic and high socialfeatures, high academic and low social features, and so on. In orderto see how the participants interacted with di�erent groups, wede�ned two kind of measurements - Diversity and Coverage. Bothof the metrics were measured by Entropy: du = −

∑4i=1 pi loд4pi ,

where pi was the probability for a particular quadrant and the pro-portion of all the user�s selection (following) [15]. �e metric ofCoverage was similar in diversity but omi�ed the quantity in eachquadrant to see how users� exploration covered all four quadrantsof the recommended items. �e Coverage metric was only com-pared between groups, due to each task only having two follow-upactivities. �e action of clicking and the amount of time spent werealso recorded in this experiment.

5 RESULTS5.1 Data AnalysisA non-parametric Wilcoxon signed-rank test of di�erence amongthe metrics was conducted and rendered a W value that indicateda signi�cant di�erence when p < 0.05. �e results are reported inTables 1 and 2. �ey indicate that the diversity (entropy) and cover-age metrics were signi�cantly di�erent among groups. �is result

Metric Group 1 (Ave.) Group 2 (Ave.) W P-valueDiversity 0.14 0.26 80 8e-06 (*)Coverage 0.29 0.53 30 0.005 (*)Click 2.58 4.11 300 0.2Time 222 173 300 0.8

Table 1: �e di�erences between the Group 1: Ranked Listand Group 2: Scatter plot. �e Diversity and Coverage met-ric re�ects how the participants “followed” the conferenceattendees. (*) means signi�cant di�erences at the 5% level(p-value < 0.05)

Metric Task 1 (Ave.) Task 2 (Ave.) W P-valueDiversity 0.20 0.21 300 0.8Click 2.24 4.52 200 0.002 (*)Time 165 228 300 0.7

Table 2: �e di�erence between the Task 1: Explore knownconnections and Task 2: Explore not known connections

supports the concept that the sca�er plot interface led the user toexplore a more diverse set of a�endees across the four quadrants.Our �ndings demonstrate the potential for showing recommenda-tions in a multidimensional view to help or change the diversityof the exploration pa�ern. We also found that the click count wassigni�cantly di�erent between tasks: it required signi�cantly morework (measured by clicks) to �nd new connections. �e data alsoshow that users in the sca�er plot group needed more clicks tocomplete their tasks (most likely because the sca�er plot did notimmediately show the participants’ names, which did appear in theranked list). At the same time, the users of the sca�er plot interfacespent less time to complete the tasks. While these di�erences arenot statistically signi�cant, it could indicate that the sca�er plotinterface helped users to quickly spot social links of interest.

5.2 Exploration PatternsFigure 3 shows the exploration parameters for the two proposedtasks performed with the interfaces. We can see a clear di�erencebetween the two interfaces. �e participants who used the rankedlist interface were more focused on the high-relevance group; i.e.items with high similarity for both the academic and social features.�e recommended items (conference a�endees) with low similarityin both recommendation features were less explored and less fol-lowed by the participants. In contrast, the participants who wereusing the sca�er plot interface showed a more diverse explorationpa�ern - they more extensively examined items with both high andlow similarities in each dimension. �is result supports the expecta-tion that the proposed sca�er plot interface helped the participantsto explore a�endees who fell outside of the ranking bias area, i.e.,items that were top ranked with a high degree of similarity.

5.3 Post-�estionnaire AnalysisFigure 4 shows the post-questionnaire analysis. We found thatboth of the interfaces had a high score on subjective feedback. Forexample, both of the interfaces were found to be informative and

Late-Breaking Results, Demonstration and Theory, Opinion & Reflection Paper UMAP’17, July 9-12, 2017, Bratislava, Slovakia

68

(a) Exploration Pa�ern on Ranked List Interface (b) Exploration Pa�ern on Sca�er plot Interface

(c) Following Pa�ern on Ranked List Interface (d) Following Pa�ern on Sca�er plot Interface

Figure 3: �e social exploration patterns of the user study. Both of the exploration and following activities are recorded fromthe “explore unknown attendees task”.

Figure 4: Usability and user satisfaction data. (*) means a signi�cant di�erence at the 5% level (p-value < 0.05)

easy to use. On average, the ranked list scored higher on ease of use,satisfaction, and intention to re-use, while the visual “sca�er plot”interface scored higher on enjoyment. However, only intention tore-use was signi�cantly di�erent between the sca�er plot interfaceand the ranked list interface. �is may be due to the fact that thesca�er plot decreased availability of detailed information such asname, a�liation, and relevance score (shown in Figure 1). �is couldalso explain why it was necessary for a user to perform more clicksto interact with recommendation items of interest. �is �ndingcon�rms the diverse exploration pa�erns on the visual interface,but it also presents the challenge of creating such an interface whilemaintaining the usability level of the baseline interface.

6 CONCLUSIONIn this paper, we proposed a new visual interface to display socialrecommendation results produced by several recommender engines.We showed that the visual interface encourages users to explorea more diverse set of recommended items. We also showed thatthe exploration pa�erns were considerably di�erent between thevisual and the baseline interfaces. �e analysis hinted that the newvisual interface required less time, but more interaction steps tocomplete the same tasks, as compared to the baseline interface.�e new visual interface had a lower score on the intention to re-use as evaluated by the user feedback. �e results indicated thatvisual interface can help users to improve exploration diversity, but

Late-Breaking Results, Demonstration and Theory, Opinion & Reflection Paper UMAP’17, July 9-12, 2017, Bratislava, Slovakia

69

that further design work would be required to match the level ofusability of the baseline system.

In the current mainstream recommender system, more a�entionis paid to the problem of presentation optimization [28]; that is, tooptimize the layout in a web page with several recommendationfunctions. We believe that this issue requires a greater level ofunderstanding on how visual e�ects can in�uence users interac-tions with a recommender system. �e interface of a recommendersystem could facilitate di�erent exploration goals. In this study,we showed that a diversity-oriented interface can help the user toexplore a more diverse set of recommended items. It sheds lighton the issue of how to design an interface for a diversity explo-ration task, but adds a challenge to maintain the system�s overallusability.

We plan to continue our work on recommender system interfacesthat help users further explore diverse recommendations. Newdesigns will maintain the recommendation model accuracy and usersatisfaction at the same levels or will show the trade-o� betweenthe metrics. �e present study does have some limitations. First, weexplored a combination of just two recommendation features. �einterface will need more work to explore more than two features.Second, the sca�er plot interface made it di�cult to recognizefamiliar names. �e user needed to click on many dots to fullyinspect the recommended results. To address this issue, we plan toexplore a combination of the sca�er plot and ranked list interfaces.

REFERENCES[1] Panagiotis Adamopoulos and Alexander Tuzhilin. 2014. On over-specialization

and concentration bias of recommendations: Probabilistic neighborhood selec-tion in collaborative �ltering systems. In Proceedings of the 8th ACM Conferenceon Recommender systems. ACM, 153–160.

[2] Gediminas Adomavicius and YoungOk Kwon. 2012. Improving aggregate rec-ommendation diversity using ranking-based techniques. IEEE Transactions onKnowledge and Data Engineering 24, 5 (2012), 896–911.

[3] Jisun An, Daniele �ercia, and Jon Crowcro�. 2013. Why individuals seekdiverse opinions (or why they don’t). In Proceedings of the 5th Annual ACM WebScience Conference. ACM, 15–18.

[4] Judit Bar-Ilan, Kevin Keenoy, Mark Levene, and Eti Yaari. 2009. Presentation biasis signi�cant in determining user preference for search results��A user study.Journal of the Association for Information Science and Technology 60, 1 (2009),135–149.

[5] Engin Bozdag and Jeroen van den Hoven. 2015. Breaking the �lter bubble:democracy and design. Ethics and Information Technology 17, 4 (2015), 249–265.

[6] Peter Brusilovsky, Jung Sun Oh, Claudia Lopez, Denis Parra, and Wei Jeng. 2016.Linking information and people in a social system for academic conferences.New Review of Hypermedia and Multimedia (2016), 1–31.

[7] Siamak Faridani, Ephrat Bi�on, Kimiko Ryokai, and Ken Goldberg. 2010. Opinionspace: a scalable tool for browsing online comments. In Proceedings of the SIGCHIConference on Human Factors in Computing Systems. ACM, 1175–1184.

[8] Giorgos Giannopoulos, Marios Koniaris, Ingmar Weber, Alejandro Jaimes, andTimos Sellis. 2015. Algorithms and criteria for diversi�cation of news articlecomments. Journal of Intelligent Information Systems 44, 1 (2015), 1–47.

[9] Eduardo Graells-Garrido, Mounia Lalmas, and Ricardo Baeza-Yates. 2016. DataPortraits and Intermediary Topics: Encouraging Exploration of Politically Di-verse Pro�les. In Proceedings of the 21st International Conference on IntelligentUser Interfaces. ACM, 228–240.

[10] Rong Hu and Pearl Pu. 2011. Helping Users Perceive Recommendation Diversity..In DiveRS@ RecSys. 43–50.

[11] Neil Hurley and Mi Zhang. 2011. Novelty and diversity in top-n recommendation–analysis and evaluation. ACM Transactions on Internet Technology (TOIT) 10, 4(2011), 14.

[12] Hannah Kim, Jaegul Choo, Haesun Park, and Alex Endert. 2016. Interaxis:Steering sca�erplot axes via observation-level interaction. IEEE transactions onvisualization and computer graphics 22, 1 (2016), 131–140.

[13] Kristina Lerman and Tad Hogg. 2014. Leveraging position bias to improve peerrecommendation. PloS one 9, 6 (2014), e98914.

[14] Q Vera Liao and Wai-Tat Fu. 2013. Beyond the �lter bubble: interactive e�ectsof perceived threat and topic involvement on selective exposure to information.

In Proceedings of the SIGCHI conference on human factors in computing systems.ACM, 2359–2368.

[15] Jennifer Moody and David H Glass. 2016. A Novel Classi�cation Frameworkfor Evaluating Individual and Aggregate Diversity in Top-N Recommendations.ACM Transactions on Intelligent Systems and Technology (TIST) 7, 3 (2016), 42.

[16] Sean A Munson and Paul Resnick. 2010. Presenting diverse political opinions:how and how much. In Proceedings of the SIGCHI conference on human factors incomputing systems. ACM, 1457–1466.

[17] Tien T Nguyen, Pik-Mai Hui, F Maxwell Harper, Loren Terveen, and Joseph AKonstan. 2014. Exploring the �lter bubble: the e�ect of using recommendersystems on content diversity. In Proceedings of the 23rd international conferenceon World wide web. ACM, 677–686.

[18] Denis Parra and Peter Brusilovsky. 2015. User-controllable personalization: Acase study with SetFusion. International Journal of Human-Computer Studies 78(2015), 43��67.

[19] Pearl Pu and Li Chen. 2007. Trust-inspiring explanation interfaces for recom-mender systems. Knowledge-Based Systems 20, 6 (2007), 542–556.

[20] Shameem A Puthiya Parambath, Nicolas Usunier, and Yves Grandvalet. 2016. Acoverage-based approach to recommendation diversity on similarity graph. InProceedings of the 10th ACM Conference on Recommender Systems. ACM, 15–22.

[21] J Ben Schafer, Joseph A Konstan, and John Riedl. 2002. Meta-recommendationsystems: user-controlled integration of diverse recommendations. In Proceedingsof the eleventh international conference on Information and knowledgemanagement.ACM, 43–51.

[22] Idan Szpektor, Yoelle Maarek, and Dan Pelleg. 2013. When relevance is notenough: promoting diversity and freshness in personalized question recommen-dation. In Proceedings of the 22nd international conference on World Wide Web.ACM, 1249–1260.

[23] Choon Hui Teo, Houssam Nassif, Daniel Hill, Sriram Srinivasan, Mitchell Good-man, Vijai Mohan, and SVN Vishwanathan. 2016. Adaptive, Personalized Diver-sity for Visual Discovery. In Proceedings of the 10th ACM Conference on Recom-mender Systems. ACM, 35–38.

[24] Chun-Hua Tsai. 2016. A Fuzzy-Based Personalized Recommender System forLocal Businesses. In Proceedings of the 27th ACM Conference on Hypertext andSocial Media. ACM, 297–302.

[25] Chun-Hua Tsai. 2017. An Interactive and Interpretable Interface for Diversity inRecommender Systems. In Proceedings of the 22Nd International Conference onIntelligent User Interfaces Companion (IUI ’17 Companion). ACM, New York, NY,USA, 225–228. DOI:h�p://dx.doi.org/10.1145/3030024.3038292

[26] Chun-Hua Tsai and Peter Brusilovsky. 2017. Providing Control and Transparencyin a Social Recommender System for Academic Conferences. In Proceedings ofthe 2017 Conference on User Modeling Adaptation and Personalization. ACM.

[27] Saul Vargas and Pablo Castells. 2011. Rank and relevance in novelty and diversitymetrics for recommender systems. In Proceedings of the ��h ACM conference onRecommender systems. ACM, 109–116.

[28] Yue Wang, Dawei Yin, Luo Jie, Pengyuan Wang, Makoto Yamada, Yi Chang, andQiaozhu Mei. 2016. Beyond ranking: Optimizing whole-page presentation. InProceedings of the Ninth ACM International Conference on Web Search and DataMining. ACM, 103–112.

[29] Jacek Wasilewski and Neil Hurley. 2016. Intent-Aware Diversi�cation Using aConstrained PLSA. In Proceedings of the 10th ACM Conference on RecommenderSystems. ACM, 39–42.

[30] David Wong, Siamak Faridani, Ephrat Bi�on, Bjorn Hartmann, and Ken Goldberg.2011. �e diversity donut: enabling participant control over the diversity ofrecommended responses. In CHI’11 Extended Abstracts on Human Factors inComputing Systems. ACM, 1471–1476.

[31] Le Wu, Qi Liu, Enhong Chen, Nicholas Jing Yuan, Guangming Guo, and Xing Xie.2016. Relevance meets coverage: A uni�ed framework to generate diversi�edrecommendations. ACM Transactions on Intelligent Systems and Technology (TIST)7, 3 (2016), 39.

[32] Wen Wu, Li Chen, and Liang He. 2013. Using personality to adjust diversity inrecommender systems. In Proceedings of the 24th ACM Conference on Hypertextand Social Media. ACM, 225–229.

[33] Cong Yu, Laks Lakshmanan, and Sihem Amer-Yahia. 2009. It takes variety tomake a world: diversi�cation in recommender systems. In Proceedings of the 12thinternational conference on extending database technology: Advances in databasetechnology. ACM, 368–378.

[34] Cong Yu, Laks VS Lakshmanan, and Sihem Amer-Yahia. 2009. Recommendationdiversi�cation using explanations. In Data Engineering, 2009. ICDE’09. IEEE 25thInternational Conference on. IEEE, 1299–1302.

[35] Tao Zhou, Zoltan Kuscsik, Jian-Guo Liu, Matus Medo, Joseph Rushton Wakeling,and Yi-Cheng Zhang. 2010. Solving the apparent diversity-accuracy dilemma ofrecommender systems. Proceedings of the National Academy of Sciences 107, 10(2010), 4511–4515.

[36] Xiaojin Zhu, Andrew B Goldberg, Jurgen Van Gael, and David Andrzejewski.2007. Improving Diversity in Ranking using Absorbing Random Walks.. InHLT-NAACL. 97–104.

Late-Breaking Results, Demonstration and Theory, Opinion & Reflection Paper UMAP’17, July 9-12, 2017, Bratislava, Slovakia

70