The role of explanations in interacting with intelligent machines
Dr Simone Stumpf Centre for Human-Computer Interaction Design [email protected] @DrSimoneStumpf
What are intelligent systems?
• Intelligent systems learn what to do by analysing user input – Learn behaviour before system is deployed (offline learning) – Update what is learned based on new data (online learning) – Take user feedback into account (interactive machine learning, active learning)
• Amazon Recommendations, Gmail Priority Inbox, Fraud detection / Network Intrusion, Clinical Decision Support Systems
• Fairly accurate but they do make mistakes – Correcting them costly, by providing more training examples
CDSS
Uncertainty
Misuse Over-Reliance
Medical observations
Suggestions
Stored Knowledge
Insufficient or incorrect:
Uncertainty
Poor data or algorithm:
Uncertainty
Uncertainty
Reliability? Trust?
Disuse Self-Reliance
or
Feedback
Why explain? • Explanations help with intelligibility (Lim & Dey 2010,
Kulesza et al. 2012)
• Explanations help with user experience (Tintarev and Mastoff 2011)
CDSS
Suggested diagnosis: Vestibular Migraine Confidence: 94% Suggested because: Patient’s Gaze test and Smooth Pursuit exam results, history of recent falls, and vertigo triggered by diet match disease profile
“Explanatory debugging” for machine learning
• Debugging relatively well understood in programming
• How can we give end users access to information that helps with debugging machine learning?
EndUser
IntelligentAgent
MentalModel
FeedbackExplanation
Effects of explanations
• Intelligibility helps with correcting machine learning (Stumpf et al. 2009, Stumpf et al. 2008)
• Explanations of system increased mental model correctness, perceived user experience and self-efficacy (Kulesza et al. 2012)
• Explanations help corrections: increased accuracy with less training data (Kulesza et al. 2015)
• Explanations considered harmful?
EndUser
IntelligentAgent
MentalModel
FeedbackExplanation
On the way to correcting machine learning
• How can we best explain system behaviour to end users? – Explanation styles – Components of explanations
• How can we support them to feedback their knowledge to the system? – Understand debugging behaviour – Ways to debug
Explanation styles
Machine learning algorithms do not work like people think they do – Complex statistical approaches – Usually not “rules”
• Switch from one “style” to another to explain an algorithm?
Explanations styles (Stumpf et al. 2009)
• Enron email dataset folders (farmer-d): Personal, Resume, Bankrupt, Enron News (122 messages)
• Lo-fi prototypes with 3 explanation styles of 3 different algorithms – Rule-based – Similarity-based – Keyword-based
• 13 participants • Think-aloud • Follows the “Natural Programming”
approach [Pane and Myers 2002]
Results
• Understanding of explanations: – Rule-based best understood – Keyword-based also good but negative weights problematic
(absence of features) – Serious understandability problems with Similarity-based
• Preference: – No clear overall winner, very individual
• Factors that mattered: – Perception of algorithm’s goodness – Clear communication of how algorithm works
Explanation components
What do we need to explain about how a machine learning system works?
– Existing Guidelines e.g. CDSS • Provide Certainty/Confidence • Provide Why?
– Intelligibility types for context-aware systems (Lime & Dey 2009; Lim 2012)
• What? Why? Why Not? What If? How to? • Inputs, Outputs, Model
• How can we explain these components best?
Results: what information is useful?
4 software versions, each with different combinations of components to explain parts of system’s behaviour, 74 participants, each used two versions of the prototype
• Impacts counts and Absence widgets not understood or liked
• Popularity Bar and Confidence well received by participants (but careful about unintended effects)
Explanation components (Kulseza et al. VLHCC13)
���
JURXQGHG� WKHRU\� PHWKRGV� >��@��� 7KH� UHVXOWLQJ� FRGH� VHW� LV�SUHVHQWHG�LQ�7DEOH�,,��
:H�WUDQVFULEHG�SDUWLFLSDQW�XWWHUDQFHV�GXULQJ�HDFK�VRQJ�DQG�DSSOLHG� WKH� FRGHV� WR� WKHVH� XWWHUDQFHV� �HDFK� FRGH� FRXOG� EH�DSSOLHG�� DW� PRVW�� RQFH� SHU� VRQJ��� 7ZR� UHVHDUFKHUV�
LQGHSHQGHQWO\�FRGHG�D�VPDOO�SRUWLRQ�RI�WKH�WUDQVFULSWV�DQG�WKHQ�GLVFXVVHG� DUHDV�RI� GLVDJUHHPHQW��2QFH� WKH� UHVHDUFKHUV� DJUHHG�RQ� WKH� FRGLQJ� RI� WKHVH� WUDQVFULSWV�� WKH\� LQGHSHQGHQWO\� FRGHG�WZR� FRPSOHWH� WUDQVFULSWV� �����RI� WKH�GDWD�²WKHLU� DJUHHPHQW��DV� FDOFXODWHG� E\� WKH� -DFFDUG� LQGH[� �WKH� LQWHUVHFWLRQ� RI� DOO�DSSOLHG�FRGHV�RYHU� WKH�XQLRQ�RI� DOO� DSSOLHG� FRGHV���ZDV������*LYHQ� WKLV� DFFHSWDEOH� OHYHO� RI� DJUHHPHQW�� D� VLQJOH� UHVHDUFKHU�FRGHG�WKH�UHPDLQLQJ�WUDQVFULSWV�DQG�SRVW�VWXG\�TXHVWLRQQDLUHV��
:H�DVVHVVHG�SDUWLFLSDQWV¶�PHQWDO�PRGHOV�XVLQJ�WKH�QXPEHU�RI� FRUUHFW� DQG� LQFRUUHFW� VWDWHPHQWV� WKH\� PDGH� GXULQJ� WKH�H[SHULPHQW�DQG�RQ�WKH�SRVW�VWXG\�TXHVWLRQQDLUH��7DEOH�,,�VKRZV�ZKLFK� W\SHV� RI� YHUEDOL]DWLRQV�UHVSRQVHV� ZHUH� FRUUHFW� YV��LQFRUUHFW��:H�WKHQ�VXPPHG�WKH�SDUWLFLSDQWV¶�FRUUHFW�UHVSRQVHV��VXEWUDFWHG�WKH�QXPEHU�RI�LQFRUUHFW�UHVSRQVHV��DQG�WUDQVODWHG�WKH�UHVXOW� WR�EH� D� VFRUH�EHWZHHQ��� �ORZHVW� ILGHOLW\�PHQWDO�PRGHO��DQG� ��� �KLJKHVW� ILGHOLW\� PHQWDO� PRGHO��� 3DUWLFLSDQWV¶�YHUEDOL]DWLRQV� GXULQJ� WKH� VWXG\� DQG� SRVW�VWXG\� TXHVWLRQQDLUH�UHVSRQVHV�ZHUH�ERWK�ZHLJKWHG�HTXDOO\��
9�� 5(68/76�
$�� 54���DQG�54����6RXQGQHVV��&RPSOHWHQHVV��DQG�7\SHV�++�SDUWLFLSDQWV�KDG�WKH�KLJKHVW�ILGHOLW\�PHQWDO�PRGHOV��$V�
)LJ�� �� VKRZV��++� SDUWLFLSDQWV� DFKLHYHG� WKUHH� RI� WKH� WRS� IRXU�VFRUHV�� ,Q�FRQWUDVW�� DOO�EXW�RQH�RI� WKH�SDUWLFLSDQWV� LQ� WKH�RWKHU�WUHDWPHQWV� FOXVWHUHG� DURXQG� ORZHU� VFRUHV�� 7KLV� VXUSULVHG� XV�EHFDXVH� ZH� KDG� H[SHFWHG� WKH� ++� WUHDWPHQW� WR� RYHUORDG�SDUWLFLSDQWV� WR� WKH� SRLQW� ZKHUH� WKH\� ZRXOG� QRW� DWWHQG� WR� VR�PXFK� FRPSOH[� LQIRUPDWLRQ�� ,QVWHDG�� ZH� H[SHFWHG� WKH� 00�WUHDWPHQW� WR� EH� D� ³VZHHW� VSRW´� LQ� WKH� WUDGH�RII� EHWZHHQ�LQIRUPDWLYHQHVV� DQG� VLPSOLFLW\²EXW� PRVW� RI� WKH� 00�SDUWLFLSDQWV�FOXVWHUHG�DURXQG�WKH�ORZHVW�VFRUHV��
)XUWKHU�� ++� SDUWLFLSDQWV¶� PHQWDO� PRGHO� VFRUHV� ZHUH�FRQVLVWHQWO\� KLJK� DFURVV� IHDWXUHV� DQG� SURFHVVHV�� DV� )LJ�� �¶V�UHVXOWV� IURP� WKH� SRVW�WDVN� TXHVWLRQQDLUH� VKRZ�� ,Q� IDFW�� DV� WKH�WKLUG� FROXPQ�RI� )LJ�� �� VKRZV��++�SDUWLFLSDQWV�ZHUH� WKH�RQO\�RQHV�WR�FRUUHFWO\�GHVFULEH�WKH�VRQJ�VHOHFWLRQ�SURFHVV��FRGHG�DV�SHU�7DEOH�,,���DQG�RQO\�RQH�++�SDUWLFLSDQW�PDGH�DQ\�LQFRUUHFW�SRVW�WDVN�REVHUYDWLRQV�DW�DOO��DV�WKH�ULJKW�KDOI�RI�)LJ����VKRZV��
���&RPSOHWHQHVV�DQG�,QWHOOLJLELOLW\�7\SHV��
7ZR� RI� WKH� LQWHOOLJLELOLW\� W\SHV�� ZK\� DQG� LQSXW�� UHODWH� WR�IHDWXUHV�� DQG�SDUWLFLSDQWV� WHQGHG� WR�GR�EHWWHU�DW�XQGHUVWDQGLQJ�
�
��
�
�
� �
�
)LJ�����([FHUSWV�IURP�WKH�:K\�WKLV�6RQJ�H[SODQDWLRQ��ZK\�LQWHOOLJLELOLW\�W\SH���7RS�OHIW��7KH�KLJK�VRXQGQHVV�VKHHW�VKRZHG�D�UDQGRP�VDPSOH�RI�GHFLVLRQ�WUHHV�IURP� WKH�EDJJLQJ� HQVHPEOH��7RS� ULJKW��(DFK� WUHH�ZDV� UHSUHVHQWHG�DV� D� VHW� RI�RUGHUHG�IHDWXUHV�ZLWK�DOORZHG�UDQJHV�RI�YDOXHV��7KH�PHGLXP�VRXQGQHVV�VKHHW�ZDV�VLPLODU��EXW�RQO\�VKRZHG�RQH�GHULYHG�GHFLVLRQ�WUHH�WKDW�DSSUR[LPDWHG�WKH�EDJJLQJ� HQVHPEOH¶V� UHDVRQLQJ�� %RWWRP� ULJKW�� 7KH� ORZ� VRXQGQHVV� VKHHW� ZDV�DOVR� VLPLODU�� EXW� RQO\� VKRZHG� RQH� GHULYHG� GHFLVLRQ� VWXPS� �VLQJOH�IHDWXUHG�WUHH��� %RWWRP� OHIW�� )RU� WKH� ++�� /6+&�� DQG�00� WUHDWPHQWV�� WKLV� VKHHW� DOVR�LQFOXGHG�WKH�FHUWDLQW\�LQWHOOLJLELOLW\�W\SH��
�)LJ�����([FHUSW�IURP�WKH�:K\�WKLV�$UWLVW�H[SODQDWLRQ��ZK\�LQWHOOLJLELOLW\�W\SH���ZKLFK�VKRZHG�WKH�DUWLVWV�VHOHFWHG�E\�WKHLU�XVHU�QHLJKERUKRRG��$OO�SDUWLFLSDQWV�UHFHLYHG�WKLV�H[SODQDWLRQ��EXW�ZLWK�GLIIHUHQW�QHLJKERUKRRG�VL]HV��VHH�7DEOH�,���
�)LJ�� ��� ([FHUSW� IURP� WKH� :KDW� WKH� &RPSXWHU� .QRZV� H[SODQDWLRQ� �LQSXW�LQWHOOLJLELOLW\� W\SH��� SURYLGLQJ� D� FRPSUHKHQVLYH� OLVW� RI� IHDWXUHV� WKDW� WKH�UHFRPPHQGHU�XVHG��$OO�SDUWLFLSDQWV�UHFHLYHG�WKLV�H[SODQDWLRQ��
�)LJ�� ��� ([FHUSW� IURP� WKH� +RZ� LW� $OO� :RUNV� 7RJHWKHU� H[SODQDWLRQ� �PRGHO�LQWHOOLJLELOLW\�W\SH���ZKLFK�VKRZHG�KRZ�WKH�SDUWLFLSDQWV¶�OLVW�RI�DUWLVWV�ZDV�XVHG�WR� PDNH� VRQJ� UHFRPPHQGDWLRQV�� �3RVLWLYH� DQG� QHJDWLYH� WUDLQLQJ� VHWV� ZHUH�FRORU�FRGHG� IRU�SDUWLFLSDQWV� WKURXJKRXW� WKH� IORZ�FKDUW���2QO\�++�DQG�/6+&�SDUWLFLSDQWV�UHFHLYHG�WKLV�H[SODQDWLRQ��
�
%UHHG�E\�1LUYDQD��
�
Why this song? 7KH�FRPSXWHU�ORRNHG�DW�ZKDW�WKH�green songs�WHQG�WR�KDYH�LQ�FRPPRQ��DQG�ZKDW�PDNHV�WKHP�GLIIHUHQW�IURP�WKH�red songs���,W�GLG�WKLV�����WLPHV��HDFK�WLPH�UDQGRPO\�SLFNLQJ�VRQJV�IURP�WKH�UHG�DQG�JUHHQ�JURXSV ���,Q�93�RI�WKH�����WLPHV��LW�SUHGLFWHG�WKDW�\RX¶OO�OLNH�WKLV�VRQJ ��+HUH�DUH�IRXU�RI�WKH�SUHGLFWLRQV��IRU�HDFK�RQH��WKH�FRPSXWHU�WKLQNV�WKDW�\RX¶OO�OLNH�VRQJV�ZLWK� all�RI�WKHVH�FKDUDFWHULVWLFV��
Prediction #1 Prediction #2
Danceability �
Danceability� �
Key and mode �
Key and mode��
Loudness � Loudness� �Beat grouping � Beat grouping� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�
Duration � Duration� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�
Energy � Energy� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�
Tempo � Tempo� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ��
Prediction #3 Prediction #4
Danceability� �
Danceability� �
Beat grouping��
Key and mode��
Key and mode��
Loudness� �Loudness� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ� Beat grouping� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�Duration� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ� Duration� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�
Energy� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ� Energy� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�Tempo� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ� Tempo� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�
Certainty
7KH�FRPSXWHU�LV�93%�FRQILGHQW�\RX¶OO�OLNH�WKLV�VRQJ��
�
�� ����27%
�� ����27%
����G% ���G%-14 dB
����G% ���G%-14 dB
���� �����2:57
�� ����89%
���ESP ����ESP159 bpm
�� ����27%
�� ����27%
����G% ���G%-14 dB
���
��
/LNH'LVOLNH
6RQJV�WKDW�VLPLODU�SHRSOH�OLVWHQHG�WR/HJHQG 6RQJV�WKDW�GLVVLPLODU�SHRSOH�OLVWHQHG�WR
%UHHG�E\�1LUYDQD��
�
Why this song? 7KH�FRPSXWHU�ORRNHG�DW�ZKDW�WKH�green songs�WHQG�WR�KDYH�LQ�FRPPRQ��DQG�ZKDW�PDNHV�WKHP�GLIIHUHQW�IURP�WKH�red songs���,W�GLG�WKLV�����WLPHV��HDFK�WLPH�UDQGRPO\�SLFNLQJ�VRQJV�IURP�WKH�UHG�DQG�JUHHQ�JURXSV ���,Q�93�RI�WKH�����WLPHV��LW�SUHGLFWHG�WKDW�\RX¶OO�OLNH�WKLV�VRQJ ��+HUH�DUH�IRXU�RI�WKH�SUHGLFWLRQV��IRU�HDFK�RQH��WKH�FRPSXWHU�WKLQNV�WKDW�\RX¶OO�OLNH�VRQJV�ZLWK� all�RI�WKHVH�FKDUDFWHULVWLFV��
Prediction #1 Prediction #2
Danceability �
Danceability� �
Key and mode �
Key and mode��
Loudness � Loudness� �Beat grouping � Beat grouping� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�
Duration � Duration� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�
Energy � Energy� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�
Tempo � Tempo� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ��
Prediction #3 Prediction #4
Danceability� �
Danceability� �
Beat grouping��
Key and mode��
Key and mode��
Loudness� �Loudness� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ� Beat grouping� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�Duration� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ� Duration� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�
Energy� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ� Energy� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�Tempo� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ� Tempo� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�
Certainty
7KH�FRPSXWHU�LV�93%�FRQILGHQW�\RX¶OO�OLNH�WKLV�VRQJ� �
�
�� ����27%
�� ����27%
����G% ���G%-14 dB
����G% ���G%-14 dB
���� �����2:57
�� ����89%
���ESP ����ESP159 bpm
�� ����27%
�� ����27%
����G% ���G%-14 dB
���
��
/LNH'LVOLNH
6RQJV�WKDW�VLPLODU�SHRSOH�OLVWHQHG�WR/HJHQG 6RQJV�WKDW�GLVVLPLODU�SHRSOH�OLVWHQHG�WR
%UHHG�E\�1LUYDQD��
�
Why this song? 7KH�FRPSXWHU�ORRNHG�DW�ZKDW�WKH�green songs�WHQG�WR�KDYH�LQ�FRPPRQ��DQG�ZKDW�PDNHV�WKHP�GLIIHUHQW�IURP�WKH�red songs���,W�GLG�WKLV�����WLPHV��HDFK�WLPH�UDQGRPO\�SLFNLQJ�VRQJV�IURP�WKH�UHG�DQG�JUHHQ�JURXSV ���
,Q�93�RI�WKH�����WLPHV��LW�SUHGLFWHG�WKDW�\RX¶OO�OLNH�WKLV�VRQJ ��+HUH�DUH�IRXU�RI�WKH�SUHGLFWLRQV��IRU�HDFK�RQH��WKH�FRPSXWHU�WKLQNV�WKDW�\RX¶OO�OLNH�VRQJV�ZLWK� all�RI�WKHVH�FKDUDFWHULVWLFV��
Prediction #1 Prediction #2
Danceability �
Danceability� �
Key and mode �
Key and mode��
Loudness � Loudness� �Beat grouping � Beat grouping� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�
Duration � Duration� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�
Energy � Energy� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�
Tempo � Tempo� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ��
Prediction #3 Prediction #4
Danceability� �
Danceability� �
Beat grouping��
Key and mode��
Key and mode��
Loudness� �Loudness� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ� Beat grouping� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�Duration� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ� Duration� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�
Energy� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ� Energy� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�Tempo� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ� Tempo� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�
Certainty
7KH�FRPSXWHU�LV�93%�FRQILGHQW�\RX¶OO�OLNH�WKLV�VRQJ��
�
�� ����27%
�� ����27%
����G% ���G%-14 dB
����G% ���G%-14 dB
���� �����2:57
�� ����89%
���ESP ����ESP159 bpm
�� ����27%
�� ����27%
����G% ���G%-14 dB
���
��
/LNH'LVOLNH
6RQJV�WKDW�VLPLODU�SHRSOH�OLVWHQHG�WR/HJHQG 6RQJV�WKDW�GLVVLPLODU�SHRSOH�OLVWHQHG�WR
,Q�<RXU�/LJKW�E\�*RW\H��
Why this song? 7KH�FRPSXWHU�ORRNHG�DW�ZKDW�WKH�green songs�WHQG�WR�KDYH�LQ�FRPPRQ��DQG�ZKDW�PDNHV�WKHP�GLIIHUHQW�IURP�WKH�red songs��,W�WKLQNV�WKDW�\RX¶OO�OLNH�VRQJV�ZLWK� all�RI�WKHVH�FKDUDFWHULVWLFV��
Danceability� �Duration 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�
Energy 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�Key and mode 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�
Loudness 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�Tempo 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�
Beat grouping 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�
Certainty
7KH�FRPSXWHU�LV�54%�FRQILGHQW�\RX¶OO�OLNH�WKLV�VRQJ��
�
�
�� ����27%
���
���
/LNH'LVOLNH
6RQJV�WKDW�VLPLODU�SHRSOH�OLVWHQHG�WR/HJHQG 6RQJV�WKDW�GLVVLPLODU�SHRSOH�OLVWHQHG�WR
�
�
How it all works together
�
<RXU�OLVW�RI�PXVLFLDQV
*HW�WKH����PRVW�SRSXODU�VRQJV�E\�HDFK�RI�WKHVH�
DUWLVWV
8VH�WKHVH�UHODWLRQVKLSV�WR�SUHGLFW�ZKHWKHU�HDFK�QHZ�VRQJ�LV�OLNH�WKH�JUHHQ�VRQJV��JRRG��RU�WKH�UHG�VRQJV��EDG�
+DYH�WKH�FRPSXWHU�ORRN�IRU�UHODWLRQVKLSV�EHWZHHQ�D�VRQJV�GHWDLOV�DQG�DQG�ZKHWKHU�LWV�LQ�WKH�JUHHQ�JURXS�RU�UHG�JURXS�
*HW�GHWDLOV�DERXW�DOO�����VRQJV
)LQG����SHRSOH�ZKR�OLVWHQ�WR�\RXU�OLVW�RI�PXVLFLDQV�6RUWHG�E\�KRZ�PDQ\�PXVLFLDQV�\RX�ERWK�OLNH�
*HW�HDFK�SHUVRQV�����PRVW�SOD\HG�VRQJV�LQ�WKH�SDVW�
\HDU
*HW�WKH����RWKHU�DUWLVWV�WKHVH�SHRSOH�PRVW�RIWHQ�
OLVWHQ�WR
��������/DVW�IP�OLVWHQHUV
)LQG����SHRSOH�ZKR�GRQW�OLVWHQ�WR�DQ\�RI�WKH�PXVLFLDQV�\RX�OLVWHG
*HW�HDFK�SHUVRQV�����PRVW�SOD\HG�VRQJV�LQ�WKH�SDVW�
\HDU
*HW�GHWDLOV�DERXW�DOO�������VRQJV
*HW�GHWDLOV�DERXW�DOO�������VRQJV
6RQJV�WKDW�VLPLODU�SHRSOH�OLVWHQHG�WR
/HJHQG
6RQJV�WKDW�GLVVLPLODU�SHRSOH�OLVWHQHG�WR
7$%/(�,�����7+(�³:+<��621*�´�,17(//,*,%,/,7<�7<3(�:$6�$9$,/$%/(�,1�$//�75($70(176��%87�,76�6281'1(66�9$5,('��7+(�27+(5�,17(//,*,%,/,7<�7<3(6�:(5(�86('�72�9$5<�&203/(7(1(66��
7UHDWPHQW� ++� 00� +6/&� /6+&��5HODWLYH�6RXQGQHVV� +LJK� 0HGLXP� +LJK� /RZ�
5HODWLYH�&RPSOHWHQHVV� +LJK� 0HGLXP� /RZ� +LJK�
,QWHOOLJLELOLW\�W\SHV�
:K\��VRQJ�� %DJJHG�GHFLVLRQ�WUHH�
'HFLVLRQ�WUHH�
%DJJHG�GHFLVLRQ�WUHH�
'HFLVLRQ�VWXPS�
:K\��DUWLVW�� 1HDUHVW�QHLJKERU��N ����
1HDUHVW�QHLJKERU��N ����
1HDUHVW�QHLJKERU��N ���
1HDUHVW�QHLJKERU��N ����
&HUWDLQW\� <HV� <HV� 1R� <HV�0RGHO� <HV� 1R� 1R� <HV�,QSXW� <HV� <HV� <HV� <HV�
�
���
JURXQGHG� WKHRU\� PHWKRGV� >��@��� 7KH� UHVXOWLQJ� FRGH� VHW� LV�SUHVHQWHG�LQ�7DEOH�,,��
:H�WUDQVFULEHG�SDUWLFLSDQW�XWWHUDQFHV�GXULQJ�HDFK�VRQJ�DQG�DSSOLHG� WKH� FRGHV� WR� WKHVH� XWWHUDQFHV� �HDFK� FRGH� FRXOG� EH�DSSOLHG�� DW� PRVW�� RQFH� SHU� VRQJ��� 7ZR� UHVHDUFKHUV�
LQGHSHQGHQWO\�FRGHG�D�VPDOO�SRUWLRQ�RI�WKH�WUDQVFULSWV�DQG�WKHQ�GLVFXVVHG� DUHDV�RI� GLVDJUHHPHQW��2QFH� WKH� UHVHDUFKHUV� DJUHHG�RQ� WKH� FRGLQJ� RI� WKHVH� WUDQVFULSWV�� WKH\� LQGHSHQGHQWO\� FRGHG�WZR� FRPSOHWH� WUDQVFULSWV� �����RI� WKH�GDWD�²WKHLU� DJUHHPHQW��DV� FDOFXODWHG� E\� WKH� -DFFDUG� LQGH[� �WKH� LQWHUVHFWLRQ� RI� DOO�DSSOLHG�FRGHV�RYHU� WKH�XQLRQ�RI� DOO� DSSOLHG� FRGHV���ZDV������*LYHQ� WKLV� DFFHSWDEOH� OHYHO� RI� DJUHHPHQW�� D� VLQJOH� UHVHDUFKHU�FRGHG�WKH�UHPDLQLQJ�WUDQVFULSWV�DQG�SRVW�VWXG\�TXHVWLRQQDLUHV��
:H�DVVHVVHG�SDUWLFLSDQWV¶�PHQWDO�PRGHOV�XVLQJ�WKH�QXPEHU�RI� FRUUHFW� DQG� LQFRUUHFW� VWDWHPHQWV� WKH\� PDGH� GXULQJ� WKH�H[SHULPHQW�DQG�RQ�WKH�SRVW�VWXG\�TXHVWLRQQDLUH��7DEOH�,,�VKRZV�ZKLFK� W\SHV� RI� YHUEDOL]DWLRQV�UHVSRQVHV� ZHUH� FRUUHFW� YV��LQFRUUHFW��:H�WKHQ�VXPPHG�WKH�SDUWLFLSDQWV¶�FRUUHFW�UHVSRQVHV��VXEWUDFWHG�WKH�QXPEHU�RI�LQFRUUHFW�UHVSRQVHV��DQG�WUDQVODWHG�WKH�UHVXOW� WR�EH� D� VFRUH�EHWZHHQ��� �ORZHVW� ILGHOLW\�PHQWDO�PRGHO��DQG� ��� �KLJKHVW� ILGHOLW\� PHQWDO� PRGHO��� 3DUWLFLSDQWV¶�YHUEDOL]DWLRQV� GXULQJ� WKH� VWXG\� DQG� SRVW�VWXG\� TXHVWLRQQDLUH�UHVSRQVHV�ZHUH�ERWK�ZHLJKWHG�HTXDOO\��
9�� 5(68/76�
$�� 54���DQG�54����6RXQGQHVV��&RPSOHWHQHVV��DQG�7\SHV�++�SDUWLFLSDQWV�KDG�WKH�KLJKHVW�ILGHOLW\�PHQWDO�PRGHOV��$V�
)LJ�� �� VKRZV��++� SDUWLFLSDQWV� DFKLHYHG� WKUHH� RI� WKH� WRS� IRXU�VFRUHV�� ,Q�FRQWUDVW�� DOO�EXW�RQH�RI� WKH�SDUWLFLSDQWV� LQ� WKH�RWKHU�WUHDWPHQWV� FOXVWHUHG� DURXQG� ORZHU� VFRUHV�� 7KLV� VXUSULVHG� XV�EHFDXVH� ZH� KDG� H[SHFWHG� WKH� ++� WUHDWPHQW� WR� RYHUORDG�SDUWLFLSDQWV� WR� WKH� SRLQW� ZKHUH� WKH\� ZRXOG� QRW� DWWHQG� WR� VR�PXFK� FRPSOH[� LQIRUPDWLRQ�� ,QVWHDG�� ZH� H[SHFWHG� WKH� 00�WUHDWPHQW� WR� EH� D� ³VZHHW� VSRW´� LQ� WKH� WUDGH�RII� EHWZHHQ�LQIRUPDWLYHQHVV� DQG� VLPSOLFLW\²EXW� PRVW� RI� WKH� 00�SDUWLFLSDQWV�FOXVWHUHG�DURXQG�WKH�ORZHVW�VFRUHV��
)XUWKHU�� ++� SDUWLFLSDQWV¶� PHQWDO� PRGHO� VFRUHV� ZHUH�FRQVLVWHQWO\� KLJK� DFURVV� IHDWXUHV� DQG� SURFHVVHV�� DV� )LJ�� �¶V�UHVXOWV� IURP� WKH� SRVW�WDVN� TXHVWLRQQDLUH� VKRZ�� ,Q� IDFW�� DV� WKH�WKLUG� FROXPQ�RI� )LJ�� �� VKRZV��++�SDUWLFLSDQWV�ZHUH� WKH�RQO\�RQHV�WR�FRUUHFWO\�GHVFULEH�WKH�VRQJ�VHOHFWLRQ�SURFHVV��FRGHG�DV�SHU�7DEOH�,,���DQG�RQO\�RQH�++�SDUWLFLSDQW�PDGH�DQ\�LQFRUUHFW�SRVW�WDVN�REVHUYDWLRQV�DW�DOO��DV�WKH�ULJKW�KDOI�RI�)LJ����VKRZV��
���&RPSOHWHQHVV�DQG�,QWHOOLJLELOLW\�7\SHV��
7ZR� RI� WKH� LQWHOOLJLELOLW\� W\SHV�� ZK\� DQG� LQSXW�� UHODWH� WR�IHDWXUHV�� DQG�SDUWLFLSDQWV� WHQGHG� WR�GR�EHWWHU�DW�XQGHUVWDQGLQJ�
�
��
�
�
� �
�
)LJ�����([FHUSWV�IURP�WKH�:K\�WKLV�6RQJ�H[SODQDWLRQ��ZK\�LQWHOOLJLELOLW\�W\SH���7RS�OHIW��7KH�KLJK�VRXQGQHVV�VKHHW�VKRZHG�D�UDQGRP�VDPSOH�RI�GHFLVLRQ�WUHHV�IURP� WKH�EDJJLQJ� HQVHPEOH��7RS� ULJKW��(DFK� WUHH�ZDV� UHSUHVHQWHG�DV� D� VHW� RI�RUGHUHG�IHDWXUHV�ZLWK�DOORZHG�UDQJHV�RI�YDOXHV��7KH�PHGLXP�VRXQGQHVV�VKHHW�ZDV�VLPLODU��EXW�RQO\�VKRZHG�RQH�GHULYHG�GHFLVLRQ�WUHH�WKDW�DSSUR[LPDWHG�WKH�EDJJLQJ� HQVHPEOH¶V� UHDVRQLQJ�� %RWWRP� ULJKW�� 7KH� ORZ� VRXQGQHVV� VKHHW� ZDV�DOVR� VLPLODU�� EXW� RQO\� VKRZHG� RQH� GHULYHG� GHFLVLRQ� VWXPS� �VLQJOH�IHDWXUHG�WUHH��� %RWWRP� OHIW�� )RU� WKH� ++�� /6+&�� DQG�00� WUHDWPHQWV�� WKLV� VKHHW� DOVR�LQFOXGHG�WKH�FHUWDLQW\�LQWHOOLJLELOLW\�W\SH��
�)LJ�����([FHUSW�IURP�WKH�:K\�WKLV�$UWLVW�H[SODQDWLRQ��ZK\�LQWHOOLJLELOLW\�W\SH���ZKLFK�VKRZHG�WKH�DUWLVWV�VHOHFWHG�E\�WKHLU�XVHU�QHLJKERUKRRG��$OO�SDUWLFLSDQWV�UHFHLYHG�WKLV�H[SODQDWLRQ��EXW�ZLWK�GLIIHUHQW�QHLJKERUKRRG�VL]HV��VHH�7DEOH�,���
�)LJ�� ��� ([FHUSW� IURP� WKH� :KDW� WKH� &RPSXWHU� .QRZV� H[SODQDWLRQ� �LQSXW�LQWHOOLJLELOLW\� W\SH��� SURYLGLQJ� D� FRPSUHKHQVLYH� OLVW� RI� IHDWXUHV� WKDW� WKH�UHFRPPHQGHU�XVHG��$OO�SDUWLFLSDQWV�UHFHLYHG�WKLV�H[SODQDWLRQ��
�)LJ�� ��� ([FHUSW� IURP� WKH� +RZ� LW� $OO� :RUNV� 7RJHWKHU� H[SODQDWLRQ� �PRGHO�LQWHOOLJLELOLW\�W\SH���ZKLFK�VKRZHG�KRZ�WKH�SDUWLFLSDQWV¶�OLVW�RI�DUWLVWV�ZDV�XVHG�WR� PDNH� VRQJ� UHFRPPHQGDWLRQV�� �3RVLWLYH� DQG� QHJDWLYH� WUDLQLQJ� VHWV� ZHUH�FRORU�FRGHG� IRU�SDUWLFLSDQWV� WKURXJKRXW� WKH� IORZ�FKDUW���2QO\�++�DQG�/6+&�SDUWLFLSDQWV�UHFHLYHG�WKLV�H[SODQDWLRQ��
�
%UHHG�E\�1LUYDQD��
�
Why this song? 7KH�FRPSXWHU�ORRNHG�DW�ZKDW�WKH�green songs�WHQG�WR�KDYH�LQ�FRPPRQ��DQG�ZKDW�PDNHV�WKHP�GLIIHUHQW�IURP�WKH�red songs���,W�GLG�WKLV�����WLPHV��HDFK�WLPH�UDQGRPO\�SLFNLQJ�VRQJV�IURP�WKH�UHG�DQG�JUHHQ�JURXSV ���,Q�93�RI�WKH�����WLPHV��LW�SUHGLFWHG�WKDW�\RX¶OO�OLNH�WKLV�VRQJ ��+HUH�DUH�IRXU�RI�WKH�SUHGLFWLRQV��IRU�HDFK�RQH��WKH�FRPSXWHU�WKLQNV�WKDW�\RX¶OO�OLNH�VRQJV�ZLWK� all�RI�WKHVH�FKDUDFWHULVWLFV��
Prediction #1 Prediction #2
Danceability �
Danceability� �
Key and mode �
Key and mode��
Loudness � Loudness� �Beat grouping � Beat grouping� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�
Duration � Duration� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�
Energy � Energy� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�
Tempo � Tempo� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ��
Prediction #3 Prediction #4
Danceability� �
Danceability� �
Beat grouping��
Key and mode��
Key and mode��
Loudness� �Loudness� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ� Beat grouping� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�Duration� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ� Duration� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�
Energy� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ� Energy� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�Tempo� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ� Tempo� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�
Certainty
7KH�FRPSXWHU�LV�93%�FRQILGHQW�\RX¶OO�OLNH�WKLV�VRQJ��
�
�� ����27%
�� ����27%
����G% ���G%-14 dB
����G% ���G%-14 dB
���� �����2:57
�� ����89%
���ESP ����ESP159 bpm
�� ����27%
�� ����27%
����G% ���G%-14 dB
���
��
/LNH'LVOLNH
6RQJV�WKDW�VLPLODU�SHRSOH�OLVWHQHG�WR/HJHQG 6RQJV�WKDW�GLVVLPLODU�SHRSOH�OLVWHQHG�WR
%UHHG�E\�1LUYDQD��
�
Why this song? 7KH�FRPSXWHU�ORRNHG�DW�ZKDW�WKH�green songs�WHQG�WR�KDYH�LQ�FRPPRQ��DQG�ZKDW�PDNHV�WKHP�GLIIHUHQW�IURP�WKH�red songs���,W�GLG�WKLV�����WLPHV��HDFK�WLPH�UDQGRPO\�SLFNLQJ�VRQJV�IURP�WKH�UHG�DQG�JUHHQ�JURXSV ���,Q�93�RI�WKH�����WLPHV��LW�SUHGLFWHG�WKDW�\RX¶OO�OLNH�WKLV�VRQJ ��+HUH�DUH�IRXU�RI�WKH�SUHGLFWLRQV��IRU�HDFK�RQH��WKH�FRPSXWHU�WKLQNV�WKDW�\RX¶OO�OLNH�VRQJV�ZLWK� all�RI�WKHVH�FKDUDFWHULVWLFV��
Prediction #1 Prediction #2
Danceability �
Danceability� �
Key and mode �
Key and mode��
Loudness � Loudness� �Beat grouping � Beat grouping� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�
Duration � Duration� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�
Energy � Energy� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�
Tempo � Tempo� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ��
Prediction #3 Prediction #4
Danceability� �
Danceability� �
Beat grouping��
Key and mode��
Key and mode��
Loudness� �Loudness� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ� Beat grouping� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�Duration� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ� Duration� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�
Energy� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ� Energy� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�Tempo� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ� Tempo� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�
Certainty
7KH�FRPSXWHU�LV�93%�FRQILGHQW�\RX¶OO�OLNH�WKLV�VRQJ� �
�
�� ����27%
�� ����27%
����G% ���G%-14 dB
����G% ���G%-14 dB
���� �����2:57
�� ����89%
���ESP ����ESP159 bpm
�� ����27%
�� ����27%
����G% ���G%-14 dB
���
��
/LNH'LVOLNH
6RQJV�WKDW�VLPLODU�SHRSOH�OLVWHQHG�WR/HJHQG 6RQJV�WKDW�GLVVLPLODU�SHRSOH�OLVWHQHG�WR
%UHHG�E\�1LUYDQD��
�
Why this song? 7KH�FRPSXWHU�ORRNHG�DW�ZKDW�WKH�green songs�WHQG�WR�KDYH�LQ�FRPPRQ��DQG�ZKDW�PDNHV�WKHP�GLIIHUHQW�IURP�WKH�red songs���,W�GLG�WKLV�����WLPHV��HDFK�WLPH�UDQGRPO\�SLFNLQJ�VRQJV�IURP�WKH�UHG�DQG�JUHHQ�JURXSV ���
,Q�93�RI�WKH�����WLPHV��LW�SUHGLFWHG�WKDW�\RX¶OO�OLNH�WKLV�VRQJ ��+HUH�DUH�IRXU�RI�WKH�SUHGLFWLRQV��IRU�HDFK�RQH��WKH�FRPSXWHU�WKLQNV�WKDW�\RX¶OO�OLNH�VRQJV�ZLWK� all�RI�WKHVH�FKDUDFWHULVWLFV��
Prediction #1 Prediction #2
Danceability �
Danceability� �
Key and mode �
Key and mode��
Loudness � Loudness� �Beat grouping � Beat grouping� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�
Duration � Duration� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�
Energy � Energy� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�
Tempo � Tempo� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ��
Prediction #3 Prediction #4
Danceability� �
Danceability� �
Beat grouping��
Key and mode��
Key and mode��
Loudness� �Loudness� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ� Beat grouping� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�Duration� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ� Duration� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�
Energy� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ� Energy� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�Tempo� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ� Tempo� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�
Certainty
7KH�FRPSXWHU�LV�93%�FRQILGHQW�\RX¶OO�OLNH�WKLV�VRQJ��
�
�� ����27%
�� ����27%
����G% ���G%-14 dB
����G% ���G%-14 dB
���� �����2:57
�� ����89%
���ESP ����ESP159 bpm
�� ����27%
�� ����27%
����G% ���G%-14 dB
���
��
/LNH'LVOLNH
6RQJV�WKDW�VLPLODU�SHRSOH�OLVWHQHG�WR/HJHQG 6RQJV�WKDW�GLVVLPLODU�SHRSOH�OLVWHQHG�WR
,Q�<RXU�/LJKW�E\�*RW\H��
Why this song? 7KH�FRPSXWHU�ORRNHG�DW�ZKDW�WKH�green songs�WHQG�WR�KDYH�LQ�FRPPRQ��DQG�ZKDW�PDNHV�WKHP�GLIIHUHQW�IURP�WKH�red songs��,W�WKLQNV�WKDW�\RX¶OO�OLNH�VRQJV�ZLWK� all�RI�WKHVH�FKDUDFWHULVWLFV��
Danceability� �Duration 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�
Energy 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�Key and mode 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�
Loudness 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�Tempo 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�
Beat grouping 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�
Certainty
7KH�FRPSXWHU�LV�54%�FRQILGHQW�\RX¶OO�OLNH�WKLV�VRQJ��
�
�
�� ����27%
���
���
/LNH'LVOLNH
6RQJV�WKDW�VLPLODU�SHRSOH�OLVWHQHG�WR/HJHQG 6RQJV�WKDW�GLVVLPLODU�SHRSOH�OLVWHQHG�WR
�
�
How it all works together
�
<RXU�OLVW�RI�PXVLFLDQV
*HW�WKH����PRVW�SRSXODU�VRQJV�E\�HDFK�RI�WKHVH�
DUWLVWV
8VH�WKHVH�UHODWLRQVKLSV�WR�SUHGLFW�ZKHWKHU�HDFK�QHZ�VRQJ�LV�OLNH�WKH�JUHHQ�VRQJV��JRRG��RU�WKH�UHG�VRQJV��EDG�
+DYH�WKH�FRPSXWHU�ORRN�IRU�UHODWLRQVKLSV�EHWZHHQ�D�VRQJV�GHWDLOV�DQG�DQG�ZKHWKHU�LWV�LQ�WKH�JUHHQ�JURXS�RU�UHG�JURXS�
*HW�GHWDLOV�DERXW�DOO�����VRQJV
)LQG����SHRSOH�ZKR�OLVWHQ�WR�\RXU�OLVW�RI�PXVLFLDQV�6RUWHG�E\�KRZ�PDQ\�PXVLFLDQV�\RX�ERWK�OLNH�
*HW�HDFK�SHUVRQV�����PRVW�SOD\HG�VRQJV�LQ�WKH�SDVW�
\HDU
*HW�WKH����RWKHU�DUWLVWV�WKHVH�SHRSOH�PRVW�RIWHQ�
OLVWHQ�WR
��������/DVW�IP�OLVWHQHUV
)LQG����SHRSOH�ZKR�GRQW�OLVWHQ�WR�DQ\�RI�WKH�PXVLFLDQV�\RX�OLVWHG
*HW�HDFK�SHUVRQV�����PRVW�SOD\HG�VRQJV�LQ�WKH�SDVW�
\HDU
*HW�GHWDLOV�DERXW�DOO�������VRQJV
*HW�GHWDLOV�DERXW�DOO�������VRQJV
6RQJV�WKDW�VLPLODU�SHRSOH�OLVWHQHG�WR
/HJHQG
6RQJV�WKDW�GLVVLPLODU�SHRSOH�OLVWHQHG�WR
7$%/(�,�����7+(�³:+<��621*�´�,17(//,*,%,/,7<�7<3(�:$6�$9$,/$%/(�,1�$//�75($70(176��%87�,76�6281'1(66�9$5,('��7+(�27+(5�,17(//,*,%,/,7<�7<3(6�:(5(�86('�72�9$5<�&203/(7(1(66��
7UHDWPHQW� ++� 00� +6/&� /6+&��5HODWLYH�6RXQGQHVV� +LJK� 0HGLXP� +LJK� /RZ�
5HODWLYH�&RPSOHWHQHVV� +LJK� 0HGLXP� /RZ� +LJK�
,QWHOOLJLELOLW\�W\SHV�
:K\��VRQJ�� %DJJHG�GHFLVLRQ�WUHH�
'HFLVLRQ�WUHH�
%DJJHG�GHFLVLRQ�WUHH�
'HFLVLRQ�VWXPS�
:K\��DUWLVW�� 1HDUHVW�QHLJKERU��N ����
1HDUHVW�QHLJKERU��N ����
1HDUHVW�QHLJKERU��N ���
1HDUHVW�QHLJKERU��N ����
&HUWDLQW\� <HV� <HV� 1R� <HV�0RGHO� <HV� 1R� 1R� <HV�,QSXW� <HV� <HV� <HV� <HV�
�
���
JURXQGHG� WKHRU\� PHWKRGV� >��@��� 7KH� UHVXOWLQJ� FRGH� VHW� LV�SUHVHQWHG�LQ�7DEOH�,,��
:H�WUDQVFULEHG�SDUWLFLSDQW�XWWHUDQFHV�GXULQJ�HDFK�VRQJ�DQG�DSSOLHG� WKH� FRGHV� WR� WKHVH� XWWHUDQFHV� �HDFK� FRGH� FRXOG� EH�DSSOLHG�� DW� PRVW�� RQFH� SHU� VRQJ��� 7ZR� UHVHDUFKHUV�
LQGHSHQGHQWO\�FRGHG�D�VPDOO�SRUWLRQ�RI�WKH�WUDQVFULSWV�DQG�WKHQ�GLVFXVVHG� DUHDV�RI� GLVDJUHHPHQW��2QFH� WKH� UHVHDUFKHUV� DJUHHG�RQ� WKH� FRGLQJ� RI� WKHVH� WUDQVFULSWV�� WKH\� LQGHSHQGHQWO\� FRGHG�WZR� FRPSOHWH� WUDQVFULSWV� �����RI� WKH�GDWD�²WKHLU� DJUHHPHQW��DV� FDOFXODWHG� E\� WKH� -DFFDUG� LQGH[� �WKH� LQWHUVHFWLRQ� RI� DOO�DSSOLHG�FRGHV�RYHU� WKH�XQLRQ�RI� DOO� DSSOLHG� FRGHV���ZDV������*LYHQ� WKLV� DFFHSWDEOH� OHYHO� RI� DJUHHPHQW�� D� VLQJOH� UHVHDUFKHU�FRGHG�WKH�UHPDLQLQJ�WUDQVFULSWV�DQG�SRVW�VWXG\�TXHVWLRQQDLUHV��
:H�DVVHVVHG�SDUWLFLSDQWV¶�PHQWDO�PRGHOV�XVLQJ�WKH�QXPEHU�RI� FRUUHFW� DQG� LQFRUUHFW� VWDWHPHQWV� WKH\� PDGH� GXULQJ� WKH�H[SHULPHQW�DQG�RQ�WKH�SRVW�VWXG\�TXHVWLRQQDLUH��7DEOH�,,�VKRZV�ZKLFK� W\SHV� RI� YHUEDOL]DWLRQV�UHVSRQVHV� ZHUH� FRUUHFW� YV��LQFRUUHFW��:H�WKHQ�VXPPHG�WKH�SDUWLFLSDQWV¶�FRUUHFW�UHVSRQVHV��VXEWUDFWHG�WKH�QXPEHU�RI�LQFRUUHFW�UHVSRQVHV��DQG�WUDQVODWHG�WKH�UHVXOW� WR�EH� D� VFRUH�EHWZHHQ��� �ORZHVW� ILGHOLW\�PHQWDO�PRGHO��DQG� ��� �KLJKHVW� ILGHOLW\� PHQWDO� PRGHO��� 3DUWLFLSDQWV¶�YHUEDOL]DWLRQV� GXULQJ� WKH� VWXG\� DQG� SRVW�VWXG\� TXHVWLRQQDLUH�UHVSRQVHV�ZHUH�ERWK�ZHLJKWHG�HTXDOO\��
9�� 5(68/76�
$�� 54���DQG�54����6RXQGQHVV��&RPSOHWHQHVV��DQG�7\SHV�++�SDUWLFLSDQWV�KDG�WKH�KLJKHVW�ILGHOLW\�PHQWDO�PRGHOV��$V�
)LJ�� �� VKRZV��++� SDUWLFLSDQWV� DFKLHYHG� WKUHH� RI� WKH� WRS� IRXU�VFRUHV�� ,Q�FRQWUDVW�� DOO�EXW�RQH�RI� WKH�SDUWLFLSDQWV� LQ� WKH�RWKHU�WUHDWPHQWV� FOXVWHUHG� DURXQG� ORZHU� VFRUHV�� 7KLV� VXUSULVHG� XV�EHFDXVH� ZH� KDG� H[SHFWHG� WKH� ++� WUHDWPHQW� WR� RYHUORDG�SDUWLFLSDQWV� WR� WKH� SRLQW� ZKHUH� WKH\� ZRXOG� QRW� DWWHQG� WR� VR�PXFK� FRPSOH[� LQIRUPDWLRQ�� ,QVWHDG�� ZH� H[SHFWHG� WKH� 00�WUHDWPHQW� WR� EH� D� ³VZHHW� VSRW´� LQ� WKH� WUDGH�RII� EHWZHHQ�LQIRUPDWLYHQHVV� DQG� VLPSOLFLW\²EXW� PRVW� RI� WKH� 00�SDUWLFLSDQWV�FOXVWHUHG�DURXQG�WKH�ORZHVW�VFRUHV��
)XUWKHU�� ++� SDUWLFLSDQWV¶� PHQWDO� PRGHO� VFRUHV� ZHUH�FRQVLVWHQWO\� KLJK� DFURVV� IHDWXUHV� DQG� SURFHVVHV�� DV� )LJ�� �¶V�UHVXOWV� IURP� WKH� SRVW�WDVN� TXHVWLRQQDLUH� VKRZ�� ,Q� IDFW�� DV� WKH�WKLUG� FROXPQ�RI� )LJ�� �� VKRZV��++�SDUWLFLSDQWV�ZHUH� WKH�RQO\�RQHV�WR�FRUUHFWO\�GHVFULEH�WKH�VRQJ�VHOHFWLRQ�SURFHVV��FRGHG�DV�SHU�7DEOH�,,���DQG�RQO\�RQH�++�SDUWLFLSDQW�PDGH�DQ\�LQFRUUHFW�SRVW�WDVN�REVHUYDWLRQV�DW�DOO��DV�WKH�ULJKW�KDOI�RI�)LJ����VKRZV��
���&RPSOHWHQHVV�DQG�,QWHOOLJLELOLW\�7\SHV��
7ZR� RI� WKH� LQWHOOLJLELOLW\� W\SHV�� ZK\� DQG� LQSXW�� UHODWH� WR�IHDWXUHV�� DQG�SDUWLFLSDQWV� WHQGHG� WR�GR�EHWWHU�DW�XQGHUVWDQGLQJ�
�
��
�
�
� �
�
)LJ�����([FHUSWV�IURP�WKH�:K\�WKLV�6RQJ�H[SODQDWLRQ��ZK\�LQWHOOLJLELOLW\�W\SH���7RS�OHIW��7KH�KLJK�VRXQGQHVV�VKHHW�VKRZHG�D�UDQGRP�VDPSOH�RI�GHFLVLRQ�WUHHV�IURP� WKH�EDJJLQJ� HQVHPEOH��7RS� ULJKW��(DFK� WUHH�ZDV� UHSUHVHQWHG�DV� D� VHW� RI�RUGHUHG�IHDWXUHV�ZLWK�DOORZHG�UDQJHV�RI�YDOXHV��7KH�PHGLXP�VRXQGQHVV�VKHHW�ZDV�VLPLODU��EXW�RQO\�VKRZHG�RQH�GHULYHG�GHFLVLRQ�WUHH�WKDW�DSSUR[LPDWHG�WKH�EDJJLQJ� HQVHPEOH¶V� UHDVRQLQJ�� %RWWRP� ULJKW�� 7KH� ORZ� VRXQGQHVV� VKHHW� ZDV�DOVR� VLPLODU�� EXW� RQO\� VKRZHG� RQH� GHULYHG� GHFLVLRQ� VWXPS� �VLQJOH�IHDWXUHG�WUHH��� %RWWRP� OHIW�� )RU� WKH� ++�� /6+&�� DQG�00� WUHDWPHQWV�� WKLV� VKHHW� DOVR�LQFOXGHG�WKH�FHUWDLQW\�LQWHOOLJLELOLW\�W\SH��
�)LJ�����([FHUSW�IURP�WKH�:K\�WKLV�$UWLVW�H[SODQDWLRQ��ZK\�LQWHOOLJLELOLW\�W\SH���ZKLFK�VKRZHG�WKH�DUWLVWV�VHOHFWHG�E\�WKHLU�XVHU�QHLJKERUKRRG��$OO�SDUWLFLSDQWV�UHFHLYHG�WKLV�H[SODQDWLRQ��EXW�ZLWK�GLIIHUHQW�QHLJKERUKRRG�VL]HV��VHH�7DEOH�,���
�)LJ�� ��� ([FHUSW� IURP� WKH� :KDW� WKH� &RPSXWHU� .QRZV� H[SODQDWLRQ� �LQSXW�LQWHOOLJLELOLW\� W\SH��� SURYLGLQJ� D� FRPSUHKHQVLYH� OLVW� RI� IHDWXUHV� WKDW� WKH�UHFRPPHQGHU�XVHG��$OO�SDUWLFLSDQWV�UHFHLYHG�WKLV�H[SODQDWLRQ��
�)LJ�� ��� ([FHUSW� IURP� WKH� +RZ� LW� $OO� :RUNV� 7RJHWKHU� H[SODQDWLRQ� �PRGHO�LQWHOOLJLELOLW\�W\SH���ZKLFK�VKRZHG�KRZ�WKH�SDUWLFLSDQWV¶�OLVW�RI�DUWLVWV�ZDV�XVHG�WR� PDNH� VRQJ� UHFRPPHQGDWLRQV�� �3RVLWLYH� DQG� QHJDWLYH� WUDLQLQJ� VHWV� ZHUH�FRORU�FRGHG� IRU�SDUWLFLSDQWV� WKURXJKRXW� WKH� IORZ�FKDUW���2QO\�++�DQG�/6+&�SDUWLFLSDQWV�UHFHLYHG�WKLV�H[SODQDWLRQ��
�
%UHHG�E\�1LUYDQD��
�
Why this song? 7KH�FRPSXWHU�ORRNHG�DW�ZKDW�WKH�green songs�WHQG�WR�KDYH�LQ�FRPPRQ��DQG�ZKDW�PDNHV�WKHP�GLIIHUHQW�IURP�WKH�red songs���,W�GLG�WKLV�����WLPHV��HDFK�WLPH�UDQGRPO\�SLFNLQJ�VRQJV�IURP�WKH�UHG�DQG�JUHHQ�JURXSV ���,Q�93�RI�WKH�����WLPHV��LW�SUHGLFWHG�WKDW�\RX¶OO�OLNH�WKLV�VRQJ ��+HUH�DUH�IRXU�RI�WKH�SUHGLFWLRQV��IRU�HDFK�RQH��WKH�FRPSXWHU�WKLQNV�WKDW�\RX¶OO�OLNH�VRQJV�ZLWK� all�RI�WKHVH�FKDUDFWHULVWLFV��
Prediction #1 Prediction #2
Danceability �
Danceability� �
Key and mode �
Key and mode��
Loudness � Loudness� �Beat grouping � Beat grouping� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�
Duration � Duration� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�
Energy � Energy� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�
Tempo � Tempo� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ��
Prediction #3 Prediction #4
Danceability� �
Danceability� �
Beat grouping��
Key and mode��
Key and mode��
Loudness� �Loudness� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ� Beat grouping� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�Duration� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ� Duration� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�
Energy� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ� Energy� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�Tempo� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ� Tempo� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�
Certainty
7KH�FRPSXWHU�LV�93%�FRQILGHQW�\RX¶OO�OLNH�WKLV�VRQJ��
�
�� ����27%
�� ����27%
����G% ���G%-14 dB
����G% ���G%-14 dB
���� �����2:57
�� ����89%
���ESP ����ESP159 bpm
�� ����27%
�� ����27%
����G% ���G%-14 dB
���
��
/LNH'LVOLNH
6RQJV�WKDW�VLPLODU�SHRSOH�OLVWHQHG�WR/HJHQG 6RQJV�WKDW�GLVVLPLODU�SHRSOH�OLVWHQHG�WR
%UHHG�E\�1LUYDQD��
�
Why this song? 7KH�FRPSXWHU�ORRNHG�DW�ZKDW�WKH�green songs�WHQG�WR�KDYH�LQ�FRPPRQ��DQG�ZKDW�PDNHV�WKHP�GLIIHUHQW�IURP�WKH�red songs���,W�GLG�WKLV�����WLPHV��HDFK�WLPH�UDQGRPO\�SLFNLQJ�VRQJV�IURP�WKH�UHG�DQG�JUHHQ�JURXSV ���,Q�93�RI�WKH�����WLPHV��LW�SUHGLFWHG�WKDW�\RX¶OO�OLNH�WKLV�VRQJ ��+HUH�DUH�IRXU�RI�WKH�SUHGLFWLRQV��IRU�HDFK�RQH��WKH�FRPSXWHU�WKLQNV�WKDW�\RX¶OO�OLNH�VRQJV�ZLWK� all�RI�WKHVH�FKDUDFWHULVWLFV��
Prediction #1 Prediction #2
Danceability �
Danceability� �
Key and mode �
Key and mode��
Loudness � Loudness� �Beat grouping � Beat grouping� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�
Duration � Duration� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�
Energy � Energy� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�
Tempo � Tempo� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ��
Prediction #3 Prediction #4
Danceability� �
Danceability� �
Beat grouping��
Key and mode��
Key and mode��
Loudness� �Loudness� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ� Beat grouping� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�Duration� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ� Duration� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�
Energy� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ� Energy� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�Tempo� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ� Tempo� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�
Certainty
7KH�FRPSXWHU�LV�93%�FRQILGHQW�\RX¶OO�OLNH�WKLV�VRQJ� �
�
�� ����27%
�� ����27%
����G% ���G%-14 dB
����G% ���G%-14 dB
���� �����2:57
�� ����89%
���ESP ����ESP159 bpm
�� ����27%
�� ����27%
����G% ���G%-14 dB
���
��
/LNH'LVOLNH
6RQJV�WKDW�VLPLODU�SHRSOH�OLVWHQHG�WR/HJHQG 6RQJV�WKDW�GLVVLPLODU�SHRSOH�OLVWHQHG�WR
%UHHG�E\�1LUYDQD��
�
Why this song? 7KH�FRPSXWHU�ORRNHG�DW�ZKDW�WKH�green songs�WHQG�WR�KDYH�LQ�FRPPRQ��DQG�ZKDW�PDNHV�WKHP�GLIIHUHQW�IURP�WKH�red songs���,W�GLG�WKLV�����WLPHV��HDFK�WLPH�UDQGRPO\�SLFNLQJ�VRQJV�IURP�WKH�UHG�DQG�JUHHQ�JURXSV ���
,Q�93�RI�WKH�����WLPHV��LW�SUHGLFWHG�WKDW�\RX¶OO�OLNH�WKLV�VRQJ ��+HUH�DUH�IRXU�RI�WKH�SUHGLFWLRQV��IRU�HDFK�RQH��WKH�FRPSXWHU�WKLQNV�WKDW�\RX¶OO�OLNH�VRQJV�ZLWK� all�RI�WKHVH�FKDUDFWHULVWLFV��
Prediction #1 Prediction #2
Danceability �
Danceability� �
Key and mode �
Key and mode��
Loudness � Loudness� �Beat grouping � Beat grouping� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�
Duration � Duration� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�
Energy � Energy� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�
Tempo � Tempo� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ��
Prediction #3 Prediction #4
Danceability� �
Danceability� �
Beat grouping��
Key and mode��
Key and mode��
Loudness� �Loudness� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ� Beat grouping� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�Duration� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ� Duration� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�
Energy� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ� Energy� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�Tempo� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ� Tempo� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�
Certainty
7KH�FRPSXWHU�LV�93%�FRQILGHQW�\RX¶OO�OLNH�WKLV�VRQJ��
�
�� ����27%
�� ����27%
����G% ���G%-14 dB
����G% ���G%-14 dB
���� �����2:57
�� ����89%
���ESP ����ESP159 bpm
�� ����27%
�� ����27%
����G% ���G%-14 dB
���
��
/LNH'LVOLNH
6RQJV�WKDW�VLPLODU�SHRSOH�OLVWHQHG�WR/HJHQG 6RQJV�WKDW�GLVVLPLODU�SHRSOH�OLVWHQHG�WR
,Q�<RXU�/LJKW�E\�*RW\H��
Why this song? 7KH�FRPSXWHU�ORRNHG�DW�ZKDW�WKH�green songs�WHQG�WR�KDYH�LQ�FRPPRQ��DQG�ZKDW�PDNHV�WKHP�GLIIHUHQW�IURP�WKH�red songs��,W�WKLQNV�WKDW�\RX¶OO�OLNH�VRQJV�ZLWK� all�RI�WKHVH�FKDUDFWHULVWLFV��
Danceability� �Duration 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�
Energy 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�Key and mode 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�
Loudness 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�Tempo 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�
Beat grouping 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�
Certainty
7KH�FRPSXWHU�LV�54%�FRQILGHQW�\RX¶OO�OLNH�WKLV�VRQJ��
�
�
�� ����27%
���
���
/LNH'LVOLNH
6RQJV�WKDW�VLPLODU�SHRSOH�OLVWHQHG�WR/HJHQG 6RQJV�WKDW�GLVVLPLODU�SHRSOH�OLVWHQHG�WR
�
�
How it all works together
�
<RXU�OLVW�RI�PXVLFLDQV
*HW�WKH����PRVW�SRSXODU�VRQJV�E\�HDFK�RI�WKHVH�
DUWLVWV
8VH�WKHVH�UHODWLRQVKLSV�WR�SUHGLFW�ZKHWKHU�HDFK�QHZ�VRQJ�LV�OLNH�WKH�JUHHQ�VRQJV��JRRG��RU�WKH�UHG�VRQJV��EDG�
+DYH�WKH�FRPSXWHU�ORRN�IRU�UHODWLRQVKLSV�EHWZHHQ�D�VRQJV�GHWDLOV�DQG�DQG�ZKHWKHU�LWV�LQ�WKH�JUHHQ�JURXS�RU�UHG�JURXS�
*HW�GHWDLOV�DERXW�DOO�����VRQJV
)LQG����SHRSOH�ZKR�OLVWHQ�WR�\RXU�OLVW�RI�PXVLFLDQV�6RUWHG�E\�KRZ�PDQ\�PXVLFLDQV�\RX�ERWK�OLNH�
*HW�HDFK�SHUVRQV�����PRVW�SOD\HG�VRQJV�LQ�WKH�SDVW�
\HDU
*HW�WKH����RWKHU�DUWLVWV�WKHVH�SHRSOH�PRVW�RIWHQ�
OLVWHQ�WR
��������/DVW�IP�OLVWHQHUV
)LQG����SHRSOH�ZKR�GRQW�OLVWHQ�WR�DQ\�RI�WKH�PXVLFLDQV�\RX�OLVWHG
*HW�HDFK�SHUVRQV�����PRVW�SOD\HG�VRQJV�LQ�WKH�SDVW�
\HDU
*HW�GHWDLOV�DERXW�DOO�������VRQJV
*HW�GHWDLOV�DERXW�DOO�������VRQJV
6RQJV�WKDW�VLPLODU�SHRSOH�OLVWHQHG�WR
/HJHQG
6RQJV�WKDW�GLVVLPLODU�SHRSOH�OLVWHQHG�WR
7$%/(�,�����7+(�³:+<��621*�´�,17(//,*,%,/,7<�7<3(�:$6�$9$,/$%/(�,1�$//�75($70(176��%87�,76�6281'1(66�9$5,('��7+(�27+(5�,17(//,*,%,/,7<�7<3(6�:(5(�86('�72�9$5<�&203/(7(1(66��
7UHDWPHQW� ++� 00� +6/&� /6+&��5HODWLYH�6RXQGQHVV� +LJK� 0HGLXP� +LJK� /RZ�
5HODWLYH�&RPSOHWHQHVV� +LJK� 0HGLXP� /RZ� +LJK�
,QWHOOLJLELOLW\�W\SHV�
:K\��VRQJ�� %DJJHG�GHFLVLRQ�WUHH�
'HFLVLRQ�WUHH�
%DJJHG�GHFLVLRQ�WUHH�
'HFLVLRQ�VWXPS�
:K\��DUWLVW�� 1HDUHVW�QHLJKERU��N ����
1HDUHVW�QHLJKERU��N ����
1HDUHVW�QHLJKERU��N ���
1HDUHVW�QHLJKERU��N ����
&HUWDLQW\� <HV� <HV� 1R� <HV�0RGHO� <HV� 1R� 1R� <HV�,QSXW� <HV� <HV� <HV� <HV�
�
���
JURXQGHG� WKHRU\� PHWKRGV� >��@��� 7KH� UHVXOWLQJ� FRGH� VHW� LV�SUHVHQWHG�LQ�7DEOH�,,��
:H�WUDQVFULEHG�SDUWLFLSDQW�XWWHUDQFHV�GXULQJ�HDFK�VRQJ�DQG�DSSOLHG� WKH� FRGHV� WR� WKHVH� XWWHUDQFHV� �HDFK� FRGH� FRXOG� EH�DSSOLHG�� DW� PRVW�� RQFH� SHU� VRQJ��� 7ZR� UHVHDUFKHUV�
LQGHSHQGHQWO\�FRGHG�D�VPDOO�SRUWLRQ�RI�WKH�WUDQVFULSWV�DQG�WKHQ�GLVFXVVHG� DUHDV�RI� GLVDJUHHPHQW��2QFH� WKH� UHVHDUFKHUV� DJUHHG�RQ� WKH� FRGLQJ� RI� WKHVH� WUDQVFULSWV�� WKH\� LQGHSHQGHQWO\� FRGHG�WZR� FRPSOHWH� WUDQVFULSWV� �����RI� WKH�GDWD�²WKHLU� DJUHHPHQW��DV� FDOFXODWHG� E\� WKH� -DFFDUG� LQGH[� �WKH� LQWHUVHFWLRQ� RI� DOO�DSSOLHG�FRGHV�RYHU� WKH�XQLRQ�RI� DOO� DSSOLHG� FRGHV���ZDV������*LYHQ� WKLV� DFFHSWDEOH� OHYHO� RI� DJUHHPHQW�� D� VLQJOH� UHVHDUFKHU�FRGHG�WKH�UHPDLQLQJ�WUDQVFULSWV�DQG�SRVW�VWXG\�TXHVWLRQQDLUHV��
:H�DVVHVVHG�SDUWLFLSDQWV¶�PHQWDO�PRGHOV�XVLQJ�WKH�QXPEHU�RI� FRUUHFW� DQG� LQFRUUHFW� VWDWHPHQWV� WKH\� PDGH� GXULQJ� WKH�H[SHULPHQW�DQG�RQ�WKH�SRVW�VWXG\�TXHVWLRQQDLUH��7DEOH�,,�VKRZV�ZKLFK� W\SHV� RI� YHUEDOL]DWLRQV�UHVSRQVHV� ZHUH� FRUUHFW� YV��LQFRUUHFW��:H�WKHQ�VXPPHG�WKH�SDUWLFLSDQWV¶�FRUUHFW�UHVSRQVHV��VXEWUDFWHG�WKH�QXPEHU�RI�LQFRUUHFW�UHVSRQVHV��DQG�WUDQVODWHG�WKH�UHVXOW� WR�EH� D� VFRUH�EHWZHHQ��� �ORZHVW� ILGHOLW\�PHQWDO�PRGHO��DQG� ��� �KLJKHVW� ILGHOLW\� PHQWDO� PRGHO��� 3DUWLFLSDQWV¶�YHUEDOL]DWLRQV� GXULQJ� WKH� VWXG\� DQG� SRVW�VWXG\� TXHVWLRQQDLUH�UHVSRQVHV�ZHUH�ERWK�ZHLJKWHG�HTXDOO\��
9�� 5(68/76�
$�� 54���DQG�54����6RXQGQHVV��&RPSOHWHQHVV��DQG�7\SHV�++�SDUWLFLSDQWV�KDG�WKH�KLJKHVW�ILGHOLW\�PHQWDO�PRGHOV��$V�
)LJ�� �� VKRZV��++� SDUWLFLSDQWV� DFKLHYHG� WKUHH� RI� WKH� WRS� IRXU�VFRUHV�� ,Q�FRQWUDVW�� DOO�EXW�RQH�RI� WKH�SDUWLFLSDQWV� LQ� WKH�RWKHU�WUHDWPHQWV� FOXVWHUHG� DURXQG� ORZHU� VFRUHV�� 7KLV� VXUSULVHG� XV�EHFDXVH� ZH� KDG� H[SHFWHG� WKH� ++� WUHDWPHQW� WR� RYHUORDG�SDUWLFLSDQWV� WR� WKH� SRLQW� ZKHUH� WKH\� ZRXOG� QRW� DWWHQG� WR� VR�PXFK� FRPSOH[� LQIRUPDWLRQ�� ,QVWHDG�� ZH� H[SHFWHG� WKH� 00�WUHDWPHQW� WR� EH� D� ³VZHHW� VSRW´� LQ� WKH� WUDGH�RII� EHWZHHQ�LQIRUPDWLYHQHVV� DQG� VLPSOLFLW\²EXW� PRVW� RI� WKH� 00�SDUWLFLSDQWV�FOXVWHUHG�DURXQG�WKH�ORZHVW�VFRUHV��
)XUWKHU�� ++� SDUWLFLSDQWV¶� PHQWDO� PRGHO� VFRUHV� ZHUH�FRQVLVWHQWO\� KLJK� DFURVV� IHDWXUHV� DQG� SURFHVVHV�� DV� )LJ�� �¶V�UHVXOWV� IURP� WKH� SRVW�WDVN� TXHVWLRQQDLUH� VKRZ�� ,Q� IDFW�� DV� WKH�WKLUG� FROXPQ�RI� )LJ�� �� VKRZV��++�SDUWLFLSDQWV�ZHUH� WKH�RQO\�RQHV�WR�FRUUHFWO\�GHVFULEH�WKH�VRQJ�VHOHFWLRQ�SURFHVV��FRGHG�DV�SHU�7DEOH�,,���DQG�RQO\�RQH�++�SDUWLFLSDQW�PDGH�DQ\�LQFRUUHFW�SRVW�WDVN�REVHUYDWLRQV�DW�DOO��DV�WKH�ULJKW�KDOI�RI�)LJ����VKRZV��
���&RPSOHWHQHVV�DQG�,QWHOOLJLELOLW\�7\SHV��
7ZR� RI� WKH� LQWHOOLJLELOLW\� W\SHV�� ZK\� DQG� LQSXW�� UHODWH� WR�IHDWXUHV�� DQG�SDUWLFLSDQWV� WHQGHG� WR�GR�EHWWHU�DW�XQGHUVWDQGLQJ�
�
��
�
�
� �
�
)LJ�����([FHUSWV�IURP�WKH�:K\�WKLV�6RQJ�H[SODQDWLRQ��ZK\�LQWHOOLJLELOLW\�W\SH���7RS�OHIW��7KH�KLJK�VRXQGQHVV�VKHHW�VKRZHG�D�UDQGRP�VDPSOH�RI�GHFLVLRQ�WUHHV�IURP� WKH�EDJJLQJ� HQVHPEOH��7RS� ULJKW��(DFK� WUHH�ZDV� UHSUHVHQWHG�DV� D� VHW� RI�RUGHUHG�IHDWXUHV�ZLWK�DOORZHG�UDQJHV�RI�YDOXHV��7KH�PHGLXP�VRXQGQHVV�VKHHW�ZDV�VLPLODU��EXW�RQO\�VKRZHG�RQH�GHULYHG�GHFLVLRQ�WUHH�WKDW�DSSUR[LPDWHG�WKH�EDJJLQJ� HQVHPEOH¶V� UHDVRQLQJ�� %RWWRP� ULJKW�� 7KH� ORZ� VRXQGQHVV� VKHHW� ZDV�DOVR� VLPLODU�� EXW� RQO\� VKRZHG� RQH� GHULYHG� GHFLVLRQ� VWXPS� �VLQJOH�IHDWXUHG�WUHH��� %RWWRP� OHIW�� )RU� WKH� ++�� /6+&�� DQG�00� WUHDWPHQWV�� WKLV� VKHHW� DOVR�LQFOXGHG�WKH�FHUWDLQW\�LQWHOOLJLELOLW\�W\SH��
�)LJ�����([FHUSW�IURP�WKH�:K\�WKLV�$UWLVW�H[SODQDWLRQ��ZK\�LQWHOOLJLELOLW\�W\SH���ZKLFK�VKRZHG�WKH�DUWLVWV�VHOHFWHG�E\�WKHLU�XVHU�QHLJKERUKRRG��$OO�SDUWLFLSDQWV�UHFHLYHG�WKLV�H[SODQDWLRQ��EXW�ZLWK�GLIIHUHQW�QHLJKERUKRRG�VL]HV��VHH�7DEOH�,���
�)LJ�� ��� ([FHUSW� IURP� WKH� :KDW� WKH� &RPSXWHU� .QRZV� H[SODQDWLRQ� �LQSXW�LQWHOOLJLELOLW\� W\SH��� SURYLGLQJ� D� FRPSUHKHQVLYH� OLVW� RI� IHDWXUHV� WKDW� WKH�UHFRPPHQGHU�XVHG��$OO�SDUWLFLSDQWV�UHFHLYHG�WKLV�H[SODQDWLRQ��
�)LJ�� ��� ([FHUSW� IURP� WKH� +RZ� LW� $OO� :RUNV� 7RJHWKHU� H[SODQDWLRQ� �PRGHO�LQWHOOLJLELOLW\�W\SH���ZKLFK�VKRZHG�KRZ�WKH�SDUWLFLSDQWV¶�OLVW�RI�DUWLVWV�ZDV�XVHG�WR� PDNH� VRQJ� UHFRPPHQGDWLRQV�� �3RVLWLYH� DQG� QHJDWLYH� WUDLQLQJ� VHWV� ZHUH�FRORU�FRGHG� IRU�SDUWLFLSDQWV� WKURXJKRXW� WKH� IORZ�FKDUW���2QO\�++�DQG�/6+&�SDUWLFLSDQWV�UHFHLYHG�WKLV�H[SODQDWLRQ��
�
%UHHG�E\�1LUYDQD��
�
Why this song? 7KH�FRPSXWHU�ORRNHG�DW�ZKDW�WKH�green songs�WHQG�WR�KDYH�LQ�FRPPRQ��DQG�ZKDW�PDNHV�WKHP�GLIIHUHQW�IURP�WKH�red songs���,W�GLG�WKLV�����WLPHV��HDFK�WLPH�UDQGRPO\�SLFNLQJ�VRQJV�IURP�WKH�UHG�DQG�JUHHQ�JURXSV ���,Q�93�RI�WKH�����WLPHV��LW�SUHGLFWHG�WKDW�\RX¶OO�OLNH�WKLV�VRQJ ��+HUH�DUH�IRXU�RI�WKH�SUHGLFWLRQV��IRU�HDFK�RQH��WKH�FRPSXWHU�WKLQNV�WKDW�\RX¶OO�OLNH�VRQJV�ZLWK� all�RI�WKHVH�FKDUDFWHULVWLFV��
Prediction #1 Prediction #2
Danceability �
Danceability� �
Key and mode �
Key and mode��
Loudness � Loudness� �Beat grouping � Beat grouping� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�
Duration � Duration� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�
Energy � Energy� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�
Tempo � Tempo� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ��
Prediction #3 Prediction #4
Danceability� �
Danceability� �
Beat grouping��
Key and mode��
Key and mode��
Loudness� �Loudness� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ� Beat grouping� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�Duration� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ� Duration� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�
Energy� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ� Energy� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�Tempo� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ� Tempo� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�
Certainty
7KH�FRPSXWHU�LV�93%�FRQILGHQW�\RX¶OO�OLNH�WKLV�VRQJ��
�
�� ����27%
�� ����27%
����G% ���G%-14 dB
����G% ���G%-14 dB
���� �����2:57
�� ����89%
���ESP ����ESP159 bpm
�� ����27%
�� ����27%
����G% ���G%-14 dB
���
��
/LNH'LVOLNH
6RQJV�WKDW�VLPLODU�SHRSOH�OLVWHQHG�WR/HJHQG 6RQJV�WKDW�GLVVLPLODU�SHRSOH�OLVWHQHG�WR
%UHHG�E\�1LUYDQD��
�
Why this song? 7KH�FRPSXWHU�ORRNHG�DW�ZKDW�WKH�green songs�WHQG�WR�KDYH�LQ�FRPPRQ��DQG�ZKDW�PDNHV�WKHP�GLIIHUHQW�IURP�WKH�red songs���,W�GLG�WKLV�����WLPHV��HDFK�WLPH�UDQGRPO\�SLFNLQJ�VRQJV�IURP�WKH�UHG�DQG�JUHHQ�JURXSV ���,Q�93�RI�WKH�����WLPHV��LW�SUHGLFWHG�WKDW�\RX¶OO�OLNH�WKLV�VRQJ ��+HUH�DUH�IRXU�RI�WKH�SUHGLFWLRQV��IRU�HDFK�RQH��WKH�FRPSXWHU�WKLQNV�WKDW�\RX¶OO�OLNH�VRQJV�ZLWK� all�RI�WKHVH�FKDUDFWHULVWLFV��
Prediction #1 Prediction #2
Danceability �
Danceability� �
Key and mode �
Key and mode��
Loudness � Loudness� �Beat grouping � Beat grouping� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�
Duration � Duration� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�
Energy � Energy� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�
Tempo � Tempo� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ��
Prediction #3 Prediction #4
Danceability� �
Danceability� �
Beat grouping��
Key and mode��
Key and mode��
Loudness� �Loudness� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ� Beat grouping� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�Duration� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ� Duration� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�
Energy� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ� Energy� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�Tempo� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ� Tempo� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�
Certainty
7KH�FRPSXWHU�LV�93%�FRQILGHQW�\RX¶OO�OLNH�WKLV�VRQJ� �
�
�� ����27%
�� ����27%
����G% ���G%-14 dB
����G% ���G%-14 dB
���� �����2:57
�� ����89%
���ESP ����ESP159 bpm
�� ����27%
�� ����27%
����G% ���G%-14 dB
���
��
/LNH'LVOLNH
6RQJV�WKDW�VLPLODU�SHRSOH�OLVWHQHG�WR/HJHQG 6RQJV�WKDW�GLVVLPLODU�SHRSOH�OLVWHQHG�WR
%UHHG�E\�1LUYDQD��
�
Why this song? 7KH�FRPSXWHU�ORRNHG�DW�ZKDW�WKH�green songs�WHQG�WR�KDYH�LQ�FRPPRQ��DQG�ZKDW�PDNHV�WKHP�GLIIHUHQW�IURP�WKH�red songs���,W�GLG�WKLV�����WLPHV��HDFK�WLPH�UDQGRPO\�SLFNLQJ�VRQJV�IURP�WKH�UHG�DQG�JUHHQ�JURXSV ���
,Q�93�RI�WKH�����WLPHV��LW�SUHGLFWHG�WKDW�\RX¶OO�OLNH�WKLV�VRQJ ��+HUH�DUH�IRXU�RI�WKH�SUHGLFWLRQV��IRU�HDFK�RQH��WKH�FRPSXWHU�WKLQNV�WKDW�\RX¶OO�OLNH�VRQJV�ZLWK� all�RI�WKHVH�FKDUDFWHULVWLFV��
Prediction #1 Prediction #2
Danceability �
Danceability� �
Key and mode �
Key and mode��
Loudness � Loudness� �Beat grouping � Beat grouping� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�
Duration � Duration� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�
Energy � Energy� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�
Tempo � Tempo� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ��
Prediction #3 Prediction #4
Danceability� �
Danceability� �
Beat grouping��
Key and mode��
Key and mode��
Loudness� �Loudness� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ� Beat grouping� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�Duration� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ� Duration� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�
Energy� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ� Energy� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�Tempo� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ� Tempo� 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�
Certainty
7KH�FRPSXWHU�LV�93%�FRQILGHQW�\RX¶OO�OLNH�WKLV�VRQJ��
�
�� ����27%
�� ����27%
����G% ���G%-14 dB
����G% ���G%-14 dB
���� �����2:57
�� ����89%
���ESP ����ESP159 bpm
�� ����27%
�� ����27%
����G% ���G%-14 dB
���
��
/LNH'LVOLNH
6RQJV�WKDW�VLPLODU�SHRSOH�OLVWHQHG�WR/HJHQG 6RQJV�WKDW�GLVVLPLODU�SHRSOH�OLVWHQHG�WR
,Q�<RXU�/LJKW�E\�*RW\H��
Why this song? 7KH�FRPSXWHU�ORRNHG�DW�ZKDW�WKH�green songs�WHQG�WR�KDYH�LQ�FRPPRQ��DQG�ZKDW�PDNHV�WKHP�GLIIHUHQW�IURP�WKH�red songs��,W�WKLQNV�WKDW�\RX¶OO�OLNH�VRQJV�ZLWK� all�RI�WKHVH�FKDUDFWHULVWLFV��
Danceability� �Duration 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�
Energy 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�Key and mode 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�
Loudness 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�Tempo 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�
Beat grouping 1RW�XVHG�IRU�WKLV�SUHGLFWLRQ�
Certainty
7KH�FRPSXWHU�LV�54%�FRQILGHQW�\RX¶OO�OLNH�WKLV�VRQJ��
�
�
�� ����27%
���
���
/LNH'LVOLNH
6RQJV�WKDW�VLPLODU�SHRSOH�OLVWHQHG�WR/HJHQG 6RQJV�WKDW�GLVVLPLODU�SHRSOH�OLVWHQHG�WR
�
�
How it all works together
�
<RXU�OLVW�RI�PXVLFLDQV
*HW�WKH����PRVW�SRSXODU�VRQJV�E\�HDFK�RI�WKHVH�
DUWLVWV
8VH�WKHVH�UHODWLRQVKLSV�WR�SUHGLFW�ZKHWKHU�HDFK�QHZ�VRQJ�LV�OLNH�WKH�JUHHQ�VRQJV��JRRG��RU�WKH�UHG�VRQJV��EDG�
+DYH�WKH�FRPSXWHU�ORRN�IRU�UHODWLRQVKLSV�EHWZHHQ�D�VRQJV�GHWDLOV�DQG�DQG�ZKHWKHU�LWV�LQ�WKH�JUHHQ�JURXS�RU�UHG�JURXS�
*HW�GHWDLOV�DERXW�DOO�����VRQJV
)LQG����SHRSOH�ZKR�OLVWHQ�WR�\RXU�OLVW�RI�PXVLFLDQV�6RUWHG�E\�KRZ�PDQ\�PXVLFLDQV�\RX�ERWK�OLNH�
*HW�HDFK�SHUVRQV�����PRVW�SOD\HG�VRQJV�LQ�WKH�SDVW�
\HDU
*HW�WKH����RWKHU�DUWLVWV�WKHVH�SHRSOH�PRVW�RIWHQ�
OLVWHQ�WR
��������/DVW�IP�OLVWHQHUV
)LQG����SHRSOH�ZKR�GRQW�OLVWHQ�WR�DQ\�RI�WKH�PXVLFLDQV�\RX�OLVWHG
*HW�HDFK�SHUVRQV�����PRVW�SOD\HG�VRQJV�LQ�WKH�SDVW�
\HDU
*HW�GHWDLOV�DERXW�DOO�������VRQJV
*HW�GHWDLOV�DERXW�DOO�������VRQJV
6RQJV�WKDW�VLPLODU�SHRSOH�OLVWHQHG�WR
/HJHQG
6RQJV�WKDW�GLVVLPLODU�SHRSOH�OLVWHQHG�WR
7$%/(�,�����7+(�³:+<��621*�´�,17(//,*,%,/,7<�7<3(�:$6�$9$,/$%/(�,1�$//�75($70(176��%87�,76�6281'1(66�9$5,('��7+(�27+(5�,17(//,*,%,/,7<�7<3(6�:(5(�86('�72�9$5<�&203/(7(1(66��
7UHDWPHQW� ++� 00� +6/&� /6+&��5HODWLYH�6RXQGQHVV� +LJK� 0HGLXP� +LJK� /RZ�
5HODWLYH�&RPSOHWHQHVV� +LJK� 0HGLXP� /RZ� +LJK�
,QWHOOLJLELOLW\�W\SHV�
:K\��VRQJ�� %DJJHG�GHFLVLRQ�WUHH�
'HFLVLRQ�WUHH�
%DJJHG�GHFLVLRQ�WUHH�
'HFLVLRQ�VWXPS�
:K\��DUWLVW�� 1HDUHVW�QHLJKERU��N ����
1HDUHVW�QHLJKERU��N ����
1HDUHVW�QHLJKERU��N ���
1HDUHVW�QHLJKERU��N ����
&HUWDLQW\� <HV� <HV� 1R� <HV�0RGHO� <HV� 1R� 1R� <HV�,QSXW� <HV� <HV� <HV� <HV�
�
Whythissong?(whyintelligibility)
Whythisar3st?(whyintelligibility)
Whatthecomputerknows?(Inputintelligibility) Howitworks?(modelintelligibility)
Results: how much information is useful?
• Low-fi prototype with pre-generated recommendations, 4 conditions, 17 participants, think aloud protocol
• Features used (inputs) better understood than process (model) • Both high completeness and soundness did better in terms of
correctness of mental models but completeness seems to matter more
• In terms of satisfaction, completeness again matters more but can trade off against soundness
• However, beware that reducing soundness may cost users’ trust in system
Principles of explanatory debugging (Kulesza et al. 2015)
• Explainability – Be iterative – Be sound – Be complete – Don’t overwhelm
• Correctability – Be actionable – Be reversible – Always honor feedback – Incremental changes matter
Explanation Components
Figure 1. The EluciDebug prototype. (A) List of folders. (B) List of messages in the selected folder. (C) The selected message.(D) Explanation of the selected message’s predicted folder. (E) Overview of which messages contain the selected word. (F) Completelist of words the learning system uses to make predictions.
them. Our EluciDebug prototype uses a multinomial naiveBayes model (MNB) [20] with feature selection [47] to meetthese constraints. Evaluating the suitability of—or changesnecessary to—other models remains an open question.
The Multinomial Naive Bayes Classifier: A Brief ReviewBefore describing how we integrated MNB with ExplanatoryDebugging, we first summarize how MNB operates. An MNBclassifier computes the probability that a given input (e.g., thedocument being classified) has of belonging to each output(e.g., the possible labels). The output with the highest prob-ability “wins” and becomes the predicted label for the input.For example, if MNB calculates that a document has a 70%probability of being junk mail and a 30% probability of notbeing junk mail, the document will be labeled as junk mail.The equations for computing probability, as defined in [20],are shown below. We use c to represent an individual classin the collection of potential output classes C, di to representan individual document to classify, and assume that the onlyfeatures the classifier uses are individual words in the set ofknown documents:
Pr(c|di) =Pr(c)Pr(di|c)
Pr(di)(1)
The term Pr(c) represents the probability that any given docu-ment belongs to class c and can be estimated by dividing thenumber of documents in c by the total number of documentsin the training set. The term Pr(di|c) represents the probability
of document di given class c and can be estimated as:
Pr(di|c) =’n
Pr(wn|c) fni (2)
The term fni is the number of instances of word n in documentdi and the term Pr(wn|c) is the probability of word n givenclass c, estimated with the equation
Pr(wn|c) =pnc+Fnc
NÂ
x=1pxc+
NÂ
x=1Fxc
(3)
where Fnc is the number of instances of word n in all of thetraining documents for class c, N is the number of uniquewords in the training documents for all classes, and pnc isa smoothing term (usually 1) to prevent the equation fromyielding 0 if no documents from class c contain word wn.The Explanatory Debugging Principles in EluciDebugBeing IterativeTo help end users iteratively build better mental models, Eluci-Debug employs two strategies: (1) each explanation focuseson an individual aspect of the learning system, and (2) layeredexplanations are available if the user wants more information.For example, the Why explanation (introduced later) primarilytells users about the system’s reasons for a specific prediction.Over time, however, the user may learn about the system’sgeneral operation by observing commonalities in the reasonsfor these specific predictions. Further, this Why explanationis layered. Part of the explanation shows users which words
The Problem…
• Explanations do not seem to matter much for “low-risk” systems (Bunt et al. 2012)
• In high-risk systems, they might have undesirable effects
Introduced to CDSS
Rate trust in CDSS
Provided clinical vignette to input into CDSS
Receive suggestion, accept or decline
Rate trust in CDSS
Comprehensive Group
Selective Group
Comprehensive
Selective
Results: confidence and explanations
Comprehensive explanations imply more knowledge and better ability of system (Over-reliance) Selective explanations suggest inferior reasoning (self-reliance)
What else do they want explained?
• Certainty/Confidence: What does it mean? How derived?
• Features and their weights
• Counter-evidence
‘So when it says the certainty is 19%, um… What does that mean?’
‘I’d want to see other things that would cause it. …So any other risk factors – patient weight, history of heart disease, history of angina?’
‘So, in medicine it’s not just the positives, it’s the negatives that you look out for as well. If I have a very clear positive, if that was the only thing I had about the patient then that’s fine, but I’d have to have the strong negatives. Only then would I accept a diagnosis.’
Further work
• Explanations largely hand-crafted to fit algorithms – No guide how to explain systems
• Effects of corrections difficult to explain – Particular components need attention: What if?
• Explanations can be overwhelming – How much is needed?
• Explanations and its effects on trust – Misuse and disuse
Conclusions
• Involving users in interactive machine learning through explanations is a viable approach
• Explanations at the moment are heavily domain and system dependent
• In a way, we haven’t gone far enough – Need better understanding of how to construct explanations – Need better understanding of effects of explanations
Top Related