Rapidly Incorporating Real Objects for Rapidly Incorporating Real Objects for Evaluation of Engineering Designs Evaluation of Engineering Designs

in a Mixed Reality Environmentin a Mixed Reality Environment

Xiyong Wang, Aaron Kotranza, Xiyong Wang, Aaron Kotranza,

John Quarles, Benjamin Lok John Quarles, Benjamin Lok Computer and Information Sciences and Computer and Information Sciences and

Engineering DepartmentEngineering DepartmentUniversity of Florida University of Florida

[xw3, akotranz, jpq, lok@cise.ufl.edu][xw3, akotranz, jpq, lok@cise.ufl.edu]

B. B. DanetteDanette Allen Allen NASA Langley Research Center NASA Langley Research Center

Danette.Allen@nasa.govDanette.Allen@nasa.gov

Driving Application: Driving Application: Engineering Design EvaluationEngineering Design Evaluation

At assembly design stageAt assembly design stage:: Several constructed parts Several constructed parts Many CAD models for parts yet to be Many CAD models for parts yet to be

constructedconstructed Unverified assembly plansUnverified assembly plans

How can we verify the assembly plan?How can we verify the assembly plan?

To be built in 2008We only have this in 2005

MotivationMotivation:: Finding design flaws earlier will save Finding design flaws earlier will save

time and moneytime and money Virtual environments have difficulty Virtual environments have difficulty

simulating hands-on assemblysimulating hands-on assembly Support team based interactionSupport team based interaction

Previous WorkPrevious Work

Engineering DesignEngineering Design Zachmann--Virtual Assembly

Simulation: Pure VE technique used in Assembly [GMD IMK, Oct 2000]

Jack – commercial software to do Jack – commercial software to do ergonomics (www.ugs.com) EDSergonomics (www.ugs.com) EDS

Interfaces in VRInterfaces in VR LaViola, Bowman (Siggraph 2001 Bowman (Siggraph 2001

course notes) – comprehensive list course notes) – comprehensive list of VR interactionof VR interaction

CollaborationCollaboration Mortensen 2002, Collaboration in Mortensen 2002, Collaboration in

tele-immersive environments. tele-immersive environments. [Eurographics][Eurographics]

Pinho 2002 Cooperative Object Pinho 2002 Cooperative Object Manipulation [VRST2002]Manipulation [VRST2002]

Image from Zachmann’s website, only the gloves (or hands) were tracked

[http://web.informatik.uni-bonn.de/II/ag-klein/people/zach/projects/fhg/ems.html]

Mixed Environment SolutionMixed Environment SolutionMixed Environments (ME):Mixed Environments (ME):

ComponentsComponentsPhysical objectsPhysical objects

CAD models CAD models

Physical correlates Physical correlates Effective InteractionEffective Interaction

Haptic feedback Haptic feedback

Visualization of Articulated modelsVisualization of Articulated models

ChallengesChallenges Capturing Capturing physical modelsphysical models Tracking Tracking real objectsreal objects Supporting Supporting teamworkteamwork

Capturing the Physical ModelCapturing the Physical ModelScanningScanning Define color markersDefine color markers

Interaction in ME

Reconstruction from Reconstruction from multiple scansmultiple scans

Tracking SolutionTracking SolutionObject tracking Object tracking

STRAPS - color trackingSTRAPS - color trackingWeb camsWeb cams

AttributesAttributes Low costLow cost Moderate latencyModerate latency Moderate accuracyModerate accuracy Many sensorsMany sensors

Limited marker spaceLimited marker space

SolutionSolution: Multi-blob tracking: Multi-blob tracking

Head trackingHead tracking Worldviz PPTWorldviz PPT Intersense Interiacube2Intersense Interiacube2

AttributesAttributes ExpensiveExpensive Low latency, Low latency, High accuracyHigh accuracy Few sensorsFew sensors

Components:Components: Real circuit boardReal circuit board Fuel tank physical correlateFuel tank physical correlate Virtual fuselage CAD Virtual fuselage CAD

modelmodel

Experiment Experiment Assembly taskAssembly task Abstracted taskAbstracted task CAD models and parts CAD models and parts

from NASAfrom NASA

Assembly Task VideoAssembly Task Video

Team Based VisualizationTeam Based VisualizationTracked Tablet PC Tracked Tablet PC For local team members For local team members

EngineersEngineers

SupervisorsSupervisors

HMDHMD For primary technicianFor primary technician

DesktopDesktop For remote observers For remote observers

Magic Lens VideoMagic Lens Video

ConclusionConclusionResultsResults

The user: The user: Found the design flawFound the design flawReordered the assembly plan Reordered the assembly plan dynamicallydynamicallyCompleted the assembly task Completed the assembly task successfullysuccessfully

Takes two hours to incorporate a Takes two hours to incorporate a real object into the MEreal object into the ME

ProblemsProblems TrackingTracking

OcclusionOcclusionLatencyLatency

Future work:Future work: Improve trackingImprove tracking

RobustnessRobustnessAdditional camerasAdditional cameras

Formal study with NASA Formal study with NASA engineersengineers

AvatarsAvatars Automate pipeline --Automate pipeline --

Reduce incorporating Reduce incorporating time to 15 minutes!!time to 15 minutes!!

More tracking camera from different location make more complicate tasks possible.

But complicated mechanism is needed to combine the information from multi channel

AcknowledgementsAcknowledgements

FundingFunding University of FloridaUniversity of Florida

PeoplePeople NASA Langley Research NASA Langley Research

CenterCenter UNCCUNCC

Dr. ShinDr. Shin

Jonathan JacksonJonathan Jackson

Dr. HodgesDr. Hodges CISE, UFCISE, UF

Robert Dickerson Robert Dickerson

Art HomsArt Homs