Student Research Week 2006 Image-based Evaluation of Video-acquired Research Skills Unmil Karadkar,...

Student Research Week 2006

Image-based Evaluation of Video-acquired Research Skills

Unmil Karadkar, Marlo Nordt

Richard Furuta

Cody Lee

Christopher Quick

Texas A&M University

Center for the Study of Digital Libraries

The Department of Computer Science

Michael E. DeBakey Institute

Veterinary Physiology & Pharmacology


Outline

• Cardiovascular Research

• Research Skill Testing Interfaces

• Design of the Evaluation

• Results and Analysis

• Future Work


Cardiovascular Research

• Related to the blood circulation system

• Involves live bats– Non-destructive, in-vivo studies– Pallid Bats – transparent wings

• Observe bat wings under a microscope

• Use these observations for research– Normal conditions– Modified conditions (occlusion, pressure)


Research Skills

• Arteries and Veins

• Order of blood vessel– Vein– Venule– Capillary

• Lymphatic vessels


Working with videos

• Single layout of images

Montage– Facilitates comparison of

consecutive images

• Motion cues

But experienced researchers

often don’t need these cues


Still Images for Testing Skills

• Validated by a pilot subject

• Properties of image layoutsSpatial organization

Temporal presentation

• Constrain or facilitate contextual informationComparison of images

– Images of blood vessels at different locations– Images of a specific location over time


Outline





• Future Work


Montage Layout

• Spatial Dimensions (0)

• Context (user dependent)

• Temporal Dimension (used)

• Slowed down video


Scrolling Layout

• Spatial Dimensions (1)

• Context (4 images)

• Temporal Dimension (used)

• Smooth scrolling


Thumbnail Layout• Spatial Dimensions (2)

• Context (all images in set)

• Temporal Dimension (unused)


Outline





• Future Work


Experimental Design 1• 3 Layouts

– Thumbnail– Scrolling– Montage– 16 images in a set

A E L

Time/image 3s 4s 2s

Answers per image per image per set

Sets 2 2 3

• 3 Tasks– Artery/Vein recognition– Size Estimation– Lymphatic Vessel Wall

Identification


Experimental Design 2• Balanced across subjects

– Ordering of tasks (AEL, ELA, LAE)– Ordering of image layouts (TMS, MST, STM)

• Powerpoint for presenting layouts

• Feedback– Verbal– Powerpoint’s Pointer feature

• Validation of answers– Panel of experts– Accurate measurements


Subjects

• Trained researchers in the bat lab– About 30 subjects available world-wide

• 9 experts– Research experience of more than 2 semesters

• 6 novices– Research experience of 1 semester only

• Age group 18 to 35

• 4 female, 11 male


Data Collection

• Answers for the tasks

• Demographic Questionnaire– Self evaluation of expertise

• Task Questionnaire– For each task

• Most preferred layout• Perceived best performance

• Short interview


Outline





• Future Work


• N=16, x=9.75 σ2=9.73• No learning• Constrained by knowledge• Preferred layout T

• N=16, x=13.59 σ2=2.13• Learning (p=0.0004)• Constrained by layout• Preferred layout T

Artery/Vein RecognitionExperts

0%

20%

40%

60%

80%

100%

T M S

Layout

Per

form

ance

Round 1

Round 2

Novices

0%

20%

40%

60%

80%

100%

T M S

Layout

Pe

rfo

rma

nc

e

Round 1

Round 2

(p=0.005)89.6%

91.4

%

Validated conventional wisdom


Size EstimationExperts

0%

20%

40%

60%

80%

M1 M2 S1 S2 T1 T2

Round

Pe

rfo

rma

nc

e +/- 40%

+/- 30%

+/- 20%

+/- 10%

Novices

0%

10%

20%

30%

40%

M1 M2 S1 S2 T1 T2

Round

Pe

rfo

rma

nce +/- 40%

+/- 30%

+/- 20%

+/- 10%

• N=16, x=9.03 σ2=14.11• Preferred layout T

• N=16, x=4.06 σ2=12.51 • Preferred layout T

General understanding of the order of blood vessels does not translate into accurate estimation of size

(p<0.002)

(p<0.002)


Lymphatic Vessel Wall Identification

Experts

0%

20%

40%

60%

80%

100%

M S T

Layout

Per

form

ance

Novices

0%

20%

40%

60%

M S T

Layout

Per

form

ance

• Worked best with layout M (p=0.035)

• Preferred layout M

(p=0.03)87%

55%

Experts and Novices use different strategies for applying contextual information

• Worked best with layout T

• Preferred layout M


Outline





• Future Work


Future Work

• Train subjects on new interfaces– Subjects preferred familiar interfaces but

performed well with the unfamiliar ones as well

• Investigate mechanisms for conveying size of objects within images– Subjects could not accurately estimate the size

of blood vessels in spite of possessing the required knowledge

• Explore other image layouts


http://ebat.tamu.edu

Center for the Study of Digital Libraries

Micahel E. DeBakey Institute

Further information

Unmil Karadkar

Marlo Nordt

Richard Furuta

[email protected]

[email protected]

[email protected]

Christopher Quick [email protected]

Student Research Week 2006 Image-based Evaluation of Video-acquired Research Skills Unmil Karadkar,...

Documents

Transcript of Student Research Week 2006 Image-based Evaluation of Video-acquired Research Skills Unmil Karadkar,...