Vis 2003 MyLifeBits Challenge for Visualization

Visualization: “Form follows function” Visualization: Science? Engineering? Or Art?

Seattle, WA22 October 2003Gordon Bell

Microsoft Research

My affair with Visualization

• 1958 wrote programs that displayed on the 32x32 scrolling display of Deuce, a machine that came from NPL that Turing worked on. Interactive Marriage proposal program.

• 1959 MIT Speech Lab and TX-0 10” 1K x 1K points, basis for DEC display. Displayed sliders, sonograms, traditional engineering graphs. Made a Land use map of Boston for Gwen Bell, a student at Harvard School of Design. They later formed the Graphics lab or such displays.

• 1960 DEC. Type 30 . Spaceware was built on it, but at one of the joint computer conferences, Ed Jacks of GM watched the display the whole time and got IBM to build them displays for car design.

• 1961 saw the difference of logic diagrams vs. logic equations; i.e. east vs west coast design

• 1986-1989 Dana>Ardent>Stardent “the graphics supercomputer”.• 1987 Commissioned the NSF Study for visualization.• 1987 FCCSET report for the Internet proposal claimed we need the net for visualization.• 1990 Keynoted first visualization conference in SF. Met the Fakespace team and

became an investor. They produced the boom for VR, large scale displays, tables, etc.

Outline• Now I find I was doing interactive vis in 1958.• Review of Vis 1990 keynote• Visualization (Google) 7M; 4.4M data• Vis 1990 then and now• Shneiderman, Tufte, & examples• Can you help MyLifeBits?

– Timelines– Document … knowledge management

1987 FCCSET Report for NREN

Titan Graphics Supercomputer c1988 25Kpolygons/sec

Keynote from Visualization I.23 October 1990

Or how I saw it then… before the web, but after NREN (Internet.)

How I "See" saw Visualization 13 years

ago* The Turing TEST for Visualization: 3D, dynamic computer

graphics and computer imaging are aimed at photo realism, ... i.e. the standard of comparison is limited by a view or photo of the phenomenon itself (reality).

The insight comes from within the user.Technology helps somewhat in the viewing.

Few of today's vizs provide the insight that compare with the creative representations that Tufte collected.

Challenge: Make the computer do what the artists did!Users need Vizicalcs (…otherwise everything is a creation)Training and integration into the computing mainstream is

still the bottleneck to use*form mental image of something not visible

Computation is the Driving Force for Visualization

4. Analysis-by-synthesis in a real time environmentReal, experimental science that simulates and simultaneously compares observations. Courtesy of Web Services

3. Simulation* is the Third Paradigm of Science & Engineering, after

2. Experimentation (taking and plotting data) and1. Theory0. Talk is for philosophers*The best way to advance a science is by building a

computer (i.e. program) for that phenomenon in an "analysis-by-synthesis" fashion.

Viz has come about through integrated computation and graphics, with a push from supercomputing.

The NSF Visualization Report Domain & Needs:

Steer simulation and computationW/S driven use of supersGraphics-oriented programming environmentsHigher-order viz of scalar, vectors ... tensor

fieldsDynamic viz of fields and flowsMassive data sets for signals and imagesAlgorithms and archs for parallel graphics & IPFramework for international Viz standards

The NSF Visualization Report (Cited) Benefits

Integrated set of portable toolsScientific progress and leadershipScientific productivityStandardization of research tools

and the cross-discipline flow of knowledge

Safeguard American Industrial Competitiveness

Making the Advanced Supercomputing facilities useful

Any Effect of the NSF Visualization Report?

Raised awareness of the opportunity for Viz by creating a word and focus

Got funding at various centers (e.g. UC/SD)Galvanized a collection of people working in

the area...Ultimately stimulated this conference* a

year or two before it might have otherwise happened

*Reaction to size and lack of applications focus at SIGGRAPH

Searching for Vizicalc:"have data, need a picture"

Users wanted to visualize their data:how much and how fast can it be displayed? how long to get the first picture?

Applications done on ad hoc case at a time basis

New tools such as AVS came about because of limited resources that had to apply the first graphics supercomputers.

Progress: 1990 - 2003• The big bang: Internet and the web

– 1987 FCCSET Report set fiber goal by 2000• Moore’s Law

– 10s of Mips >> 1-4 Gips; Custom rendering…– 10s of Mbytes >> 100 Gbytes– 25 K polygons/sec >> 25 M polygons/sec– Mpixels = Mpixels; mono; but lower cost– 1.5 Mbps > 1-2 Gbps (peak)

• Progress in Vis– Realistic rendering… see complete virtual scenes/sets– Expensive, large screens and caves:

• Using “2001, A Space Odyssey” progress in:– Visualization was much, much greater– AI, including speech was much, much less

Princeton 6K x 3K+ pixel; 18’x8’; $100K

Vis 2003

Problem and opportunity:

have many nouns…

Need:create verbs

Taxonomy • Pseudo phenomena (ICs, molecules, colored

physical parameters e.g. T, p, fields for flow, stress• Photo realism

– Real world: movies, VR, Augmented Reality,– People and digital immortality– Telepresence… ever emerging, potentially important

• Data visualization… Tufte gallery candidates?– Small no. of dimensions– Large dimensionality information maps

• Ordinary text and user visualization beyond UI

Shneiderman’s insights “exploratory data analysis and visualization process is …

characterized by mantra for visual information seeking:

“Overview, zoom and filter, details-on-demand”

“The future … larger, higher resolution screens, that present perceptually-rich & information-abundant displays.

… the flood of information can be turned into a river of knowledge...

Human perceptual skills are quite remarkable and largely underutilized”

Tufte on Graphic Excellence

Tufte

Tufte Napoleon Map Paris-Moscow

Data maps… near and dear

Network flow of trade

Hillis Taxonomy

Axon

Kiviat/radar of car parameters

Evolution of the “Ideal State” at Each Stage: Entrepreneurials

Business PlanManufacturing

ProductDevelopment

Technology/Engineering

Control

FundabilityCash

Board of Directors

Team

CEO

Sales

MarketingStage III.Product Development

Stage II.Seed

Stage I.Concept

Stage IV.Market Development

Many dimensions.. Inverse pattern recognition

Tufte: On PowerPoint

I am data

The guinea pig

• Gordon Bell is digitizing his life• Has scanned/encoded virtually all:

– All computer files– Personal documents (memos and email,

bills, legal documents, papers written, …)– Books written (and read when possible)– Photos– Posters, paintings, photo of things (artifacts, …medals, plaques)– CD collection– Home movies and videos

• Now recording real time: phone, radio, TV (movies), web pages… started meetings, conversations to come

• Paperless throughout 2002. 12” scanned, 12’ discarded.• Only 30 GB!!!

The MyLifeBits Challenge

• Files, photos, documents• Challenge is viz relevant for organizing,

searching, and utilizing your shoebox… data problem

• Tufte vs. Visioneers: – Visualization is an art. – Visualization is science and it has to be automatic!

• Challenge: – 1. What are the questions viz can help answer? – 2. What viz can help make a life or after life interesting?

• We have many nouns. You make verbs.

Statistics of use

Visualization Needs & Possibilities

• Time lines• Video cliplets, photos searching• Automatic story creation from content• Related document maps• Concept mapping of entire space• Collections & taxonomies• Synthesis of talking heads “in their words”• Tweening from photos to recreate entire movies

of the experience.

Personal LifeLog Applications

Conservator

Baby Book

Companion Caretaker

Babysitter

AdvisorMentorTutor

Autobiography

Photo Album

PersonalAssistant

Diary/Journal

Biography

Financial Manager

Medical Manager

ExecutorObituary

Others Self

Assistant for Elderly

Application controlled by:

Oth

ers

Sel

fA

pp

lica

tio

n u

sed

by:

Personal Proxy

Parole Officer

Pers Flight Recorder

Meeting Prep

Captain’s Log

Trustee

InternetMyLifeBits

store

database

files

Voice annotation tool

Text annotation tool

Legacy applications

MAPI interface

Legacy email client

Radio EPG tool

PocketPC transfer tool

Telephone capture tool

Radio capture tool

TV capture tool

TV EPG download tool

Browser tool

PocketRadio player

MyLifeBits Shell

MyLifeBits organization: time and space

Timeline/Context(space)

Personal(some $s)

GB Co.(angel, etc.)

ProfessionalACM, etc., …@Microsoft.com,

New co’s.

Archival (time) Working

1900 1910 1920 1930 1940 1950 1960 1970 1980 1990 2000 2010

C,L m d d

CGB... GB SR mB,L KF SB

Where KvMO B ABos P B WCa

6-year --GS-HS--MIT DEC---+++++.+++---++++

Education KV-----mit,F cmu

Work Bell Elec DECcmuDEC E,NSF MSFT

ComputerMuseum M B SiValley

Books BN SBN HiTechVent

Computers 4-6 11 VAX E T

Awards..

Time-machine ComputingRekimoto

Lifestreams: View by timeDavid Gelernter et al

Rock & Roll History graph

Timeline view tells a story

MyLifeBit v2 clustered time view

…Information Visualization within a Digital Video Library

Colored notches show Colored notches show where matches occurwhere matches occur

…Information Visualization within a Digital Video Library

Keyword color Keyword color matches color in matches color in rankingranking

Here are the photos

Getting the user to tell a story is the ultimate in media value

A story is a “layout” in time and spaceA story is a “layout” in time and space Most valuable content (by selection, and by being well annotated)Most valuable content (by selection, and by being well annotated) Stories must include links to any media they use (for future navigation/search – Stories must include links to any media they use (for future navigation/search –

“transclusion”).“transclusion”). Cf: MovieMaker; Creative Memories PhotoAlbumsCf: MovieMaker; Creative Memories PhotoAlbums

Dapeng was an Dapeng was an intern at BARC intern at BARC for the summer for the summer of 2000of 2000

We took him to We took him to lunch at our lunch at our favorite Dim Sum favorite Dim Sum place to say place to say farewellfarewell

At table L-R: Dapeng, Gordon, Tom, Jim, Don, At table L-R: Dapeng, Gordon, Tom, Jim, Don, Vicky, Patrick, JimVicky, Patrick, Jim

Digital Immortality: CMU Interactive Einstein:

U of MD display of health data

http://www.research.ibm.com/history/gallery.htm Document Changes History Flow

Temporal Analytics - ThemeRiver

Challenge: identify Challenge: identify patterns, trends, and patterns, trends, and relationships over time relationships over time in large data in large data collections collections

Breakthrough:Breakthrough: show show change over time change over time using smooth, stacked using smooth, stacked curves to enhance curves to enhance understanding; understanding; include context of include context of external events. external events.

Tn

Haystack: Ozone interface screenshot

PortalPortal Agents Agents

construct each construct each windowwindow

Information Visualization within a Digital Video Library

Christel & Martin (CMU), Christel & Martin (CMU), Information Visualization Information Visualization within a Digital Video within a Digital Video Library, Intelligent Library, Intelligent Information Systems, Information Systems, June 1998.June 1998.

Inxight VizServer

http://www.inxight.com/map/

More inxight

60 Business Sensitive

Engineered for broad deploymentNow under pilot evaluation and deploymentWindows based: client/server, stand alone, or web based

61 Business Sensitive

Towards Multiple Simultaneous Hypothesis Queries

JapanProtectionMeasures

Japan

Trade

Protection

Measures

TradeProtection

TradeProtectionMeasures

Each Hypothesis on a Ray

Challenge: Multiple hypothesis

analysis

Breakthrough: Interactive and visual

method for analysis of multiple hypothesis

Impact: First example of

hypothesis and scenario based analytics

Tn

Visualizing document relationshipsBoyack, Wylie & Davidson

WebScout (MSR Cambridge)

Clustered browsingof the Fine Arts Museum of San Francisco

http://www.cs.berkeley.edu/~kobus/research/projects/browse.htmlhttp://www.cs.berkeley.edu/~kobus/research/projects/browse.html

The brain

The Axon Idea processor

..\..\Cyberall MyLifeBits\Axon Idea Processor.mht

Mind mapping

Innolab 3D File manager

Webmap

Webmap

http://www.webmap.com/trademapdemo.html

Smartmoney.com

Smartmoney.com Whole market

The end…