Attention Allocation Aid for Visual Search - · PDF fileAttention Allocation Aid for...

Click here to load reader

  • date post

  • Category


  • view

  • download


Embed Size (px)

Transcript of Attention Allocation Aid for Visual Search - · PDF fileAttention Allocation Aid for...

  • Attention Allocation Aid for Visual SearchArturo Deza

    Dynamical NeuroscienceUC Santa Barbara

    [email protected]

    Jeffrey R. PetersMechanical Engineering

    UC Santa [email protected]

    Grant S. TaylorU.S. Army

    Aviation DevelopmentDirectorate, USA

    [email protected]

    Amit SuranaUnited Technologies Research

    Center, [email protected]

    Miguel P. EcksteinPsychological and Brain

    Sciences, UC Santa [email protected]

    ABSTRACTThis paper outlines the development and testing of a novel,feedback-enabled attention allocation aid (AAAD), which usesreal-time physiological data to improve human performancein a realistic sequential visual search task. Indeed, by optimiz-ing over search duration, the aid improves efficiency, whilepreserving decision accuracy, as the operator identifies andclassifies targets within simulated aerial imagery. Specifically,using experimental eye-tracking data and measurements abouttarget detectability across the human visual field, we developfunctional models of detection accuracy as a function of searchtime, number of eye movements, scan path, and image clutter.These models are then used by the AAAD in conjunction withreal time eye position data to make probabilistic estimations ofattained search accuracy and to recommend that the observereither move on to the next image or continue exploring thepresent image. An experimental evaluation in a scenario moti-vated from human supervisory control in surveillance missionsconfirms the benefits of the AAAD.

    ACM Classification KeywordsH.1.2 [Models and Principles]: User/Machine SystemsHuman factors, Human information processing; H.4.2 [In-formation Systems Applications]: Types of SystemsDecisionSupport; H.m [Miscellaneous]; I.6.4 [Simulation and Model-ing]: Model Validation and Analysis; General Terms: Experi-mentation, Human Factors, Verification

    Author KeywordsAttention; cognitive load; decision making; visual search.

    0All authors of the paper are also affiliated with the Institute forCollaborative Biotechnologies.

    Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for components of this work owned by others than ACMmust be honored. Abstracting with credit is permitted. To copy otherwise, or republish,to post on servers or to redistribute to lists, requires prior specific permission and/or afee. Request permissions from [email protected] 2017, May 6-11, 2017, Denver, CO, USA.Copyright 2017 ACM ISBN 978-1-4503-4655-9/17/05 ...$15.00.

    INTRODUCTIONThe maturation of visual sensor technology has steadily in-creased the amount of real-time data that is available in modernsurveillance mission scenarios ranging across military, home-land security and commercial applications. In many cases,it is the job of a human operator to ensure that this data isprocessed quickly and accurately. For example, supervisorysystems involving collaboration between human operators andunmanned vehicles often require the sequential processing ofimagery that is generated by the autonomous vehicles on-board cameras for the purpose of finding targets, analyzingterrain, and making key planning decisions [39]. The incredi-ble volume of data generated by modern sensors, combinedwith the complex nature of modern mission scenarios, makesoperators susceptible to information overload and attentionallocation inefficiencies [9], which can lead to detrimental per-formance and potentially dire consequences [44]. As such, thedevelopment of tools to improve human performance in visualdata analysis tasks is crucial to ensuring mission success.

    This article focuses on the development and experimental ver-ification of a novel, attention allocation aid that is designed tohelp human operators in a sequential visual search task, whichrequires the detection and classification of targets within a sim-ulated landscape. Our study is primarily motivated by surveil-lance applications that require humans to find high value tar-gets within videos generated by remote sensors, e.g., mountedon unmanned vehicles; however, the presented method is ap-plicable to a variety of application domains.

    Specifically, the main contribution of this paper is the introduc-tion and experimental verification of a real-time and feedback-enabled attention allocation aid (AAAD), which optimizesthe operators decision speed when they are engaging in targetsearch, without sacrificing performance. The motivating ob-servation is that humans have imperfect awareness of the timerequired to acquire all task-relevant visual information duringsearch, and thus are generally inefficient at administering theirtime when scrutinizing the data sets. The proposed aid makesreal-time automated search duration recommendations basedon three key metrics: 1) visual search time, 2) number ofeye movements executed by the observer, and 3) an estimated








    ] 1

    4 Ja

    n 20


  • target detectability based on prior measurements of both thetarget visibility across the visual field and the observers fixa-tions during search. In particular, these metrics are used by theaid to estimate the time required for the operator to acquire thevisual information that is necessary to support the search deci-sions, and subsequently indicate when this time has elapsedvia a simple indicator on the user interface. We experimen-tally evaluate the AAAD in a simulated surveillance scenariomotivated by human supervisory control, and found a factorof1.5 increase in user efficiency in detecting and classifyingtargets in realistic visual imagery from a slowly moving sensor.The AAAD pipeline is generic and can readily be extendedand applied to other sources of images, i.e., satellite images,astronomical images [6], x-rays for medical imaging [1, 13]or security scanning [5], and video surveillance [45] whichinclude a human-in-the-loop [53, 52].

    Our rigorous development of the AAAD also includes a num-ber of secondary contributions, including: definition of de-tectability surfaces based on eye-tacking measurements, in-corporation of image clutter effects, creation of a compositeexploration map, and utilization of a probabilistic frameworkfor decision making when computing overall search satisfac-tion based on time, eye movements, and detectability scores.

    PREVIOUS WORKResearch in human computer interaction has capitalized onbasic vision science research by including samples of the thevisual environment through eye movements (active vision)within a larger modeling framework that includes cognitive,motor, and perceptual processing involved in visual search(Tseng and Howes, 2015 [48] and Halverson and Hofson,2011 [20] which expands on the work of Kieras and Meyer,1997 [32]). Such models have been proposed for potential usefor computer interface evaluation and design.

    Another line of research has focused on how to augment hu-man capabilities in coordinating multiple tasks. For example,models have been used to optimize how observers split theirattentional resources when simultaneously conducting twodifferent visuo-cognitive tasks [36].

    Attention allocation aids have been studied in the context ofhuman supervisory control of large data acquired by multipleautomated agents (e.g., [46, 9]). Such scenarios present thechallenge of a human having to inspect large data sets withpossible errors due to visual limitations, attentional bottle-necks, and fatigue. The use of advanced physiological sensingthrough eye-tracking technology has become a viable optionfor both the assessment of the operator cognitive state, andthe evaluation of operator performance in a number of real-istic applications, e.g. [49]. One line of research attempts touse eye-tracking measurements to detect physiological andcognitive precursors to behavior such as perceived workload,fatigue, or situational awareness. Indeed, objective measuressuch as blink rates [51], pupil diameter [50, 31] , and fixa-tion/saccade characteristics [2], all have correlations to cog-nitive processing, although the use of such measurements asreliable indicators of operator mental states is not fully under-stood [12]. If undesirable states can be accurately anticipatedwith physiological measures, then they can be used to drive

    automated aids that mediate operator resources through, e.g.,optimization of task schedules [38] or adaptive automationschemes [22, 43].

    Other researchers have utilized eye tracker technology to showincreased efficiency of human search by relying on a divideand conquer collaborative search [57, 7]. In such schemes,multiple observers (usually two) engage in target search simul-taneously with real-time updates of their partners gaze andaccess to voice communication.

    A novel approach investigated in the current work is the designof an attention allocation aid that uses eye-tracking data tomake real time inferences of the attained target detection ac-curacy and critically, the time to achieve asymptotic accuracy.Such estimates, which we will refer to as search satisfactiontime, are utilized by the attention allocation aid to recommendthat the user end the current search and move on to the nextdata set. In addition, if the observer completes search priorto the search satisfaction time, the eye position data can alsobe utilized to assess whether some area of the image remainsunexplored, and suggest that the observer to further explorethat area.

    The success of the proposed approach requires an adequate un-derstanding of the relation between fixational