Visualizing and Interacting with Diamond Jason I. Hong March 14 2006.

Visualizing and Interacting with Diamond

Jason I. Hong

March 14 2006

Overview

• Interaction techniques for Diamond• Integrated Diamond Environment• Metrics for evaluating Diamond UIs

• None of what I will talk about has been built• Lots of feedback will be useful

Information Visualization

• Problem: How to view lots of data simultaneously?• Traditionally done through information visualization

0

200

400

600

800

1000

1200

A M J J A S O N D J F M

Prior 12 months

Latest 12 months

Information Visualization


Idea #1: Summarize as Metadata


– These kinds of views good for seeing trends and relations

– Views are also interactive

• Observation: Showing pictures (ie search results) makes it hard to see any trends and relations

• Idea: Diamond should focus first on showing metadata rather than pictures

Example: Stanford Photo Browser

Idea #1: Summarize as MetadataDiscussion

• Faceted metadata– Can explore categories in any order

– Ex. Location then time, or time then weather then location

• Provides feedforward (like feedback)– United States (1822)

– Gives an idea if drilling down into a category is a good idea

• So how might this work for Diamond?

Idea #1: Summarize as MetadataRough Interface Mockup #1

People0 people (2135…)1 person (1029…)2 people (613…)3 people (323…)4+ people (705…)

Cars0 cars (1036)1 car (1664)2 cars (1324)3 cars (1156)4+ cars (1422)

GrassNo grass (1603)Some grass (1400)Lots of grass (1311)

Search Filters

Search Results

Photo Metadata

http://cnn.com/

http://cnn.com/

http://cnn.com/

http://cnn.com/

http://cnn.com/

http://cnn.com/

http://cnn.com/

http://cnn.com/

http://cnn.com/

http://cnn.com/

http://cnn.com/

http://cnn.com/

http://cnn.com/


People0 people (1635)1 person (829)2 people (213)3 people (123)4+ people (305)

Cars (all > 1 car)


Photo Metadata Search Filters

Search Results (1664 results)

1 Car (1664 results)

http://cnn.com/

http://cnn.com/

http://cnn.com/

http://cnn.com/

http://cnn.com/

http://cnn.com/

http://cnn.com/

http://cnn.com/

http://cnn.com/

http://images.google.com/imgres?imgurl=http://www.ihpva.org/incoming/2001/car-small.jpg&imgrefurl=http://www.ihpva.org/incoming/2001/&h=324&w=432&sz=109&tbnid=XFC9dLrTekgivM:&tbnh=92&tbnw=123&hl=en&start=2&prev=/images%3Fq%3Dcar%26svnum%3D10%26hl%3Den%26lr%3D%26sa%3DN

http://images.google.com/imgres?imgurl=http://www.atpm.com/9.03/california/images/car.jpg&imgrefurl=http://www.atpm.com/9.03/california/car.shtml&h=1200&w=1600&sz=284&tbnid=gYHOAQQcZ4wvEM:&tbnh=112&tbnw=150&hl=en&start=12&prev=/images%3Fq%3Dcar%26svnum%3D10%26hl%3Den%26lr%3D%26sa%3DN

http://images.google.com/imgres?imgurl=http://www.highland-wedding-car.co.uk/Brodie%2520P1jpg.jpg&imgrefurl=http://www.highland-wedding-car.co.uk/&h=535&w=955&sz=57&tbnid=K-plxt1Q6F54IM:&tbnh=82&tbnw=147&hl=en&start=47&prev=/images%3Fq%3Dcar%26start%3D40%26svnum%3D10%26hl%3Den%26lr%3D%26sa%3DN

http://images.google.com/imgres?imgurl=http://users.tmok.com/%7Efirebane/car/fleft.jpg&imgrefurl=http://users.tmok.com/%7Efirebane/car/&h=600&w=800&sz=135&tbnid=vtHa_LAUtjphAM:&tbnh=106&tbnw=142&hl=en&start=60&prev=/images%3Fq%3Dcar%26start%3D40%26svnum%3D10%26hl%3Den%26lr%3D%26sa%3DN

http://images.google.com/imgres?imgurl=http://www.unitycars.com/unity_cars.jpg&imgrefurl=http://www.unitycars.com/&h=420&w=560&sz=27&tbnid=PCMMLuLT23IKrM:&tbnh=98&tbnw=131&hl=en&start=64&prev=/images%3Fq%3Dcar%26start%3D60%26svnum%3D10%26hl%3Den%26lr%3D%26sa%3DN

http://images.google.com/imgres?imgurl=http://www.sandsmuseum.com/cars/elise/thecar/side.jpg&imgrefurl=http://www.sandsmuseum.com/cars/elise/thecar/&h=335&w=504&sz=17&tbnid=WQ7vaqOxNoogXM:&tbnh=85&tbnw=128&hl=en&start=79&prev=/images%3Fq%3Dcar%26start%3D60%26svnum%3D10%26hl%3Den%26lr%3D%26sa%3DN


People (all > 1 person)

Cars (all > 1 car)


Photo Metadata Search Filters

Search Results (829 results)

1 Car (1664 results)

1 Person (829 results)

http://cnn.com/

http://cnn.com/

http://cnn.com/

http://cnn.com/

http://cnn.com/

http://images.google.com/imgres?imgurl=http://www.burtmountain.com/images/New%2520Car.jpg&imgrefurl=http://www.burtmountain.com/toc.htm&h=1600&w=1200&sz=423&tbnid=eFChqXrsV2f3cM:&tbnh=150&tbnw=112&hl=en&start=120&prev=/images%3Fq%3Dcar%26start%3D100%26svnum%3D10%26hl%3Den%26lr%3D%26sa%3DN

http://images.google.com/imgres?imgurl=http://www.lri.fr/%7Emezaour/pictures/audi_playstation_car_behind_me.JPG&imgrefurl=http://www.lri.fr/%7Emezaour/pictures/slides/audi_playstation_car_behind_me.html&h=2048&w=1536&sz=614&tbnid=buMG8uKHO4myHM:&tbnh=150&tbnw=112&hl=en&start=143&prev=/images%3Fq%3Dcar%26start%3D140%26svnum%3D10%26hl%3Den%26lr%3D%26sa%3DN

http://images.google.com/imgres?imgurl=http://www.cs.cmu.edu/%7Edesney/photographs/CarTrouble/desney_car.jpg&imgrefurl=http://www.cs.cmu.edu/%7Edesney/photographs/CarTrouble/&h=378&w=504&sz=41&tbnid=Bh8B3xZQQg9QnM:&tbnh=96&tbnw=128&hl=en&start=181&prev=/images%3Fq%3Dcar%26start%3D180%26svnum%3D10%26hl%3Den%26lr%3D%26sa%3DN

http://images.google.com/imgres?imgurl=http://korns.org/unknowns/Aquilla-Barnes-w-car-25KB.jpg&imgrefurl=http://korns.org/whats_new.htm&h=721&w=516&sz=25&tbnid=INILV6tN-YZ6TM:&tbnh=139&tbnw=99&hl=en&start=334&prev=/images%3Fq%3Dcar%26start%3D320%26svnum%3D10%26hl%3Den%26lr%3D%26sa%3DN


People

Cars

Grass

Search Results Search Filters

28,438 photos

Server 1

53,129 photos

Server 2

10,298 photos

Server 3

Add Filter

This side gives an overview of how many results there are, where pictures are coming from

Two-headed scrollbars let you easily select a range of values rather than just a single one (as in previous UI)


People

Cars

Grass

Search Results Search Filters

Server 1

Server 2

Server 3

Add Filter

28,438 photos

53,129 photos

10,298 photos

1,352 photos


• Metadata approach good for exploratory tasks– When you want to get a feel for the data set

– Probably suboptimal for confirmatory tasks

• Digression: Exploratory vs Confirmatory tasks– Exploratory more about understanding the data

• Let the data suggest interesting questions• Interactive overviews useful here

– Confirmatory more about validating a hypothesis• Use data to prove something true or false

– Diamond needs to support both kinds of tasks


• Metadata approach good for stovepiping– Some organizations don’t want to or can’t share all info

– Legal or market reasons

– Metadata approach a way of sharing enough useful info without giving everything away

– Ex. FBI, CIA, NSA, let them explore each other’s metadata, and if probable cause, then examine relevant pictures (and then log it for accountability)


• Issue: how to get good labels for metadata?– Semantic gap between people and vision algorithms

– Each filter can be a label, but not often useful• RGB values not useful as metadata here

• Issue: confidence intervals for metadata?– Ex. 90% sure there are 3 people in this image

Idea #2: Scatter / Gather for Clustering

• Problem: Lots of ways of ways of organizing pictures, how to do so in a flexible and meaningful way?

• Example: 2000 books on European History– Could organize by author, by nation, by era, political, etc

– But hard to do this in advance if you aren’t sure what categories are going to be useful

• Scatter / Gather developed at PARC for text retrieval– System scatters documents into a number of clusters

– User can gather clusters together

– System re-scatters


• Retrieve top 250 documents containing “star”– From an encyclopedia

Idea #2: Scatter / Gather for ClusteringNo semantics a priori, just some clustering algorithms to

group things by similarity


• Can we apply Scatter / Gather to Diamond?– Easier to cluster documents (vector of words) than images

• Text is simple, images have “hidden” content

– Scale and performance also issues

• Currently unclear how effective it would be– Open research question

– Need lots of help here from vision and learning people


• Digression: Different kinds of tools seem more useful for different stages of use in Diamond

• Opening– Coarse-grained, cull out obvious misses

– Exploration and metadata overview

– Query specification

• Midgame– Refine and drill-down

– Scatter/Gather

• Endgame– Fine-grained

http://images.google.com/imgres?imgurl=http://www.tqnyc.org/NYC040800/chess-smxx_03.jpg&imgrefurl=http://www.tqnyc.org/NYC040800/&h=318&w=294&sz=6&tbnid=Pugt9fdVXe26pM:&tbnh=113&tbnw=104&hl=en&start=31&prev=/images%3Fq%3Dchess%2B%26start%3D20%26svnum%3D10%26hl%3Den%26lr%3D%26sa%3DN

Idea #3: Why and Why Not Questions

• Problem: It can be confusing as to why a certain picture is shown (or not shown)

• Example: search for “faces”




• Whyline– reduced novice programmers' debugging time ~7.8 and

helping them get 40% further through their tasks


• Possible alternative interface for relevance feedback– Relevance feedback often a black box

– ‘Why’ and ‘why not’ questions can expose more info for developing and tuning filters

• Why questions– Why do my filters say there is a face here?

– Why is this image in this category?

– Also, how to fix?

• Why not questions– Harder to ask, may not know image is missing

– More useful with known data set

Overview

• Interaction techniques for Diamond• Integrated Diamond Environment• Metrics for evaluating Diamond UIs

Idea #4: Integrated Diamond Environment

• Problem: How to develop new user interfaces?– Discussed lots of UI ideas

– Don’t want to custom build a new UI for each domain

– UI toolkits, but not quite right, lots of duplicated effort

• Alternative idea: An Integrated Diamond Environment– Like Integrated Developer Environment for programming

Idea #4: Integrated Diamond EnvironmentEclipse IDE

• A single UI for managing all tasks related to development

• Comes with large set of tools• Supports plug-ins for new tools• End-users can easily re-arrange tools (windows) to fit their needs


• Provides standard tools for manipulating data sets• Advantages

– Better re-use of code– Fosters a developer community (network effects)– Foster more deployment of Diamond?

• Setting up a UI for new domain would consist of:– setting up the image filters– setting up the windows– creating useful macros


• Issue: Need a common way of integrating Diamond data flow and new tools

• Idea: Provide abstract notion of data set, plus belt of tools for viewing and manipulating

Current Tool

Tool

Tool

ToolData Sets

History


People0 people (2135…)1 person (1029…)2 people (613…)3 people (323…)4+ people (705…)


Scatter / GatherTool

Set ManipulationTool

Tool

Working Data Set (5326)

Metadata Overview

Start (5326)

http://cnn.com/

http://cnn.com/

http://cnn.com/

http://cnn.com/

http://cnn.com/

http://cnn.com/

http://cnn.com/

http://cnn.com/

http://cnn.com/

http://cnn.com/


People (all > 1 person)Cars

0 cars (1036)1 car (1664)2 cars (1324)3 cars (1156)4+ cars (1422)

Scatter / GatherTool


Tool


Metadata Overview

Start (5326) 1 person (1029)

http://cnn.com/

http://cnn.com/

http://cnn.com/

http://cnn.com/

http://cnn.com/

http://cnn.com/


Yellow, Door, Grass

Face

MetadataOverview Tool


Tool


Scatter / Gather Tool

Start (5326) 1 person (1029)







Yellow, Door, Grass

Face



Tool

Pics with 1 Person (1029)


Start (5326)








Yellow, Door, Grass

Face



Tool

Pics with 1 Person (1029)


Start (5326)


Tool A (629) Tool B (200)


• Data Set– Abstract set of images + History of how it got there

• Tool– Manipulates current data set– Provides view of that data set– Developers create these


• Macros– Would hide some of the tools– Automate repetitive actions– Describe things in a domain-specific manner– UI designers and advanced end-users create these

• Collection of tools, loosely joined– Easy to add new tools w/o a lot of overhead– Easy to assemble tools into an ensemble– Like Unix philosophy


• Idea: Model-based user interfaces– Given an abstract task model, generate a high quality

user interface with domain-specific terminology

– Alternative to macros described earlier

– Task model might include things like:• Kinds of images interested in• Higher-level description of objects in images

– Lots of green near bottom of image is “grass”

• Workflow (first do A, then do B)

– Model-based UIs researched for 20+ years• Diamond might be constrained enough though

Idea #5: Metrics

• How to measure the effectiveness of Diamond UIs?– Developing a new Diamond UI

– Using a Diamond UI

Idea #5: MetricsMethod #1 – Standard HCI task-based approach

• Define standard tasks– Ex. 1000 pictures

– Warmup task, find any picture that has “1 car + 1 person”

– Medium task (1-2 pictures with answer), see if a tractor trailer ever passed through this gate

– Harder tasks

• Measures– Time on task

– #errors

– #tasks successfully completed

– Outside judges measure quality of work

– Subjective satisfaction

• This would be a standard way of evaluating a UI

Idea #5: MetricsMethod #1 – Standard HCI task-based approach

• Problem: Diamond intended for use by experts whose time is very valuable, making it hard to go this route– Hard to get lots of experts

– Hard to get statistically significant results

– Unclear what UI to compare against (only useful if there is another UI that does something like Diamond)

• Not clear to me how to address this problem– Usually we would try to find good approximate users,

ex. medical students instead of doctors

– Focus here likely to be qualitative rather than quantitative

Idea #5: MetricsMethod #2 – TREC-like Bakeoff

• TREC stands for Text REtrieval Conference

“Existing research groups (such as the Cornell SMART system) report a doubling in performance over the six years of TREC, whereas systems new to TREC typically double their performance in the first year as they move their techniques into current state-of-the-art.”

American Society for Information Science Bulletin, Apr/May 98


• TREC worked because it targeted the generic problem of text retrieval first rather than specific problems– Initially focused on searching generic text corpora– Only recently has there been a genomics track, legal track,

blogs, video, terabyte storage, etc

• With Diamond, we are focusing on specific apps first– Pharmaceutical, Medical, Homeland security– Right from an interaction design standpoint, might not be

from an optimization standpoint


• Idea: Push for a new track in TREC for large-scale image retrieval

• Idea: Focus on a domain that will have LOTS of users– Improvements in the general case will also help specific cases

• Idea: Provide a standard large corpora of images– Bettman archive, AP photo archive, web image search– Provide an open Diamond server for others– Foster competitions on top of Diamond– Open source code + “open source” data– Student could just download code and run it

Summary

• Interaction techniques for Diamond– Faceted metadata for overviews

– Scatter / Gather for refining

– Why and Why Not questions for debugging

• Integrated Diamond Environment– A framework for developing tools and UIs

• Metrics for evaluating Diamond UIs– Standard human-computer interaction evaluation

– TREC-like bakeoffs

Backup slides

Idea: Summarize as MetadataRough Interface Mockup

People0 people (2135)1 person (1029)2 people (613)3 people (323)4+ people (705)



http://cnn.com/

http://cnn.com/

http://cnn.com/

http://cnn.com/

http://cnn.com/

http://cnn.com/

http://cnn.com/

http://cnn.com/

http://cnn.com/

http://cnn.com/

http://cnn.com/

http://cnn.com/

http://cnn.com/


http://images.google.com/imgres?imgurl=http://www.atpm.com/9.03/california/images/car.jpg&imgrefurl=http://www.atpm.com/9.03/california/car.shtml&h=1200&w=1600&sz=284&tbnid=gYHOAQQcZ4wvEM:&tbnh=112&tbnw=150&hl=en&start=12&prev=/images%3Fq%3Dcar%26svnum%3D10%26hl%3Den%26lr%3D%26sa%3DN

http://images.google.com/imgres?imgurl=http://www.highland-wedding-car.co.uk/Brodie%2520P1jpg.jpg&imgrefurl=http://www.highland-wedding-car.co.uk/&h=535&w=955&sz=57&tbnid=K-plxt1Q6F54IM:&tbnh=82&tbnw=147&hl=en&start=47&prev=/images%3Fq%3Dcar%26start%3D40%26svnum%3D10%26hl%3Den%26lr%3D%26sa%3DN



http://images.google.com/imgres?imgurl=http://www.sandsmuseum.com/cars/elise/thecar/side.jpg&imgrefurl=http://www.sandsmuseum.com/cars/elise/thecar/&h=335&w=504&sz=17&tbnid=WQ7vaqOxNoogXM:&tbnh=85&tbnw=128&hl=en&start=79&prev=/images%3Fq%3Dcar%26start%3D60%26svnum%3D10%26hl%3Den%26lr%3D%26sa%3DN

http://images.google.com/imgres?imgurl=http://www.jbl.com/car/featured/installs/solara_pics/large/DSCN0032.jpg&imgrefurl=http://www.jbl.com/car/featured/installs/default.asp&h=488&w=650&sz=100&tbnid=0aN6SlfoUK2vrM:&tbnh=101&tbnw=135&hl=en&start=96&prev=/images%3Fq%3Dcar%26start%3D80%26svnum%3D10%26hl%3Den%26lr%3D%26sa%3DN


Other Diamond Issues

• Sketch Retrieval– Draw what you want and retrieve

– Spatial as well

• Spatial retrieval– Car in corner

• Handling false negatives• Multiple people evaluating search results• Model-based user interfaces

– Given an abstract task model, generate a high quality user interface with domain-specific terminology

Idea: Other Domains for Diamond

• Digital Libraries• Web search

Visualizing and Interacting with Diamond Jason I. Hong March 14 2006.

Documents

Transcript of Visualizing and Interacting with Diamond Jason I. Hong March 14 2006.