PARK - Tangible Augmented Prototyping of Digital Handheld Products

download PARK - Tangible Augmented Prototyping of Digital Handheld Products

of 12

Transcript of PARK - Tangible Augmented Prototyping of Digital Handheld Products

  • 8/10/2019 PARK - Tangible Augmented Prototyping of Digital Handheld Products

    1/12

    Tangible augmented prototyping of digital handheld products

    Hyungjun Park a ,*, Hee-Cheol Moon a , Jae Yeol Lee ba Department of Industrial Engineering, Chosun University, 375 Seosuk-dong, Dong-gu, Gwangju 501-759, South Koreab Department of Industrial Engineering, Chonnam National University, 300 Yongbong-dong, Buk-gu, Gwangju 500-757, South Korea

    1. Introduction

    For most digital handheld products such as a mobile phone andan MP3 player, their functional behavior is very complicated andnearly all expressed as humanmachine interaction (HMI) tasks,each of which may trigger the transition between the states of theproducts. For successful entry of a new product into thecompetitive world market, it is imperative to reduce time tomarketas much as possible while precisely converting its demandsinto actual product forms, features, and functions [1,2] . Anessential activity required is the efcient and extensive use of prototypes during the product development process [1,2] .

    With recent advances in computer technology, virtual proto-typing (VP) has been considered as a new and powerfulprototyping solution to overcome the shortcomings of conven-tional prototyping methods. The concept of VP has been widelyemployed and implemented in many industrial elds includingautomotive and airplane industries [7,8] , but most works havebeen based on using virtual reality (VR) techniques [310] , andthey have been focused on visualization [2,9] , assembly and

    disassembly testing [1012] , manufacturing process simulation[13,14] , structural analysis [2,6] , and ergonomic analysis [2,9] .Some works have been conducted on capturing and simulating thefunctional behaviors of digital handheld products in VP applica-tions [15,16] . In VR-based prototyping solutions, it is not easy tobuild a virtual environment of ne quality (e.g. making detailedand realistic three-dimensional models) and to acquire tangibleuser interaction with low cost VR devices. Recently, augmentedreality (AR) approaches have been applied as alternatives fordeveloping VP solutions to overcome these shortcomings [1723] .

    In order to realize faithfully the virtual design and prototypingof digital handheld products such as mobile phones and MP3players, it is very important to provide the people involved inproduct development with tangible user interaction, the realisticvisualization of the products, and the vivid simulation of theirfunctional behaviors in a virtual environment. In this paper, wepropose a novelapproach to virtual prototyping of digital handheldproducts, which can satisfy such requirements by combining AR-based tangible interaction with functional behavior simulation.We call it tangible augmented prototyping .

    The proposed approach does not require high-cost devices suchas data gloves and haptic devices for user interaction. Rapidprototyping (RP) and paper-based modeling are properly adoptedin building AR-based tangible objects whose manipulationin an AR

    Computers in Industry 60 (2009) 114125

    A R T I C L E I N F O

    Article history:

    Received 3 December 2007Received in revised form 4 July 2008Accepted 6 September 2008Available online 17 December 2008

    Keywords:Virtual prototypingTangible objectsAugmented realityUser interactionFunctional behavior simulation

    A B S T R A C T

    Proposed in this paper is a novel approach to virtual prototyping of digital handheld products using

    augmented reality (AR)-based tangible interaction and functional behavior simulation. For tangible userinteraction in an AR environment, we use twotypesof tangible objects: oneis fora product, andthe otheris for a pointer. The user can create input events by touching specied regions of the product-typetangible object with the pointer-type tangible object. Rapid prototyping and paper-based modeling areadopted to fabricate the AR-based tangible objects which play an important role in improving theaccuracy and tangibility of user interaction. For functional behavior simulation, we adopt a statetransition methodology to capture thefunctional behavior of the product into an information model, andbuild a nite state machine (FSM) to control the transition between states of the product based on theinformation model. The FSM is combined with the AR-based tangible objects whose operations in the AR environment facilitate the tangible interaction, realistic visualization and functional simulation of adigital handheldproduct.Based on theproposed approach,a prototyping systemhas been developedandapplied for the design evaluation of various digital handheld products with encouraging feedback fromusers.

    2008 Elsevier B.V. All rights reserved.

    * Corresponding author. Tel.: +82 62 230 7039; fax: +82 62 230 7128.E-mail address: [email protected] (H. Park).

    Contents lists available at ScienceDirect

    Computers in Industry

    j ou rna l homepage : www.e l sev i e r. com/ loca t e / compind

    0166-3615/$ see front matter 2008 Elsevier B.V. All rights reserved.

    doi: 10.1016/j.compind.2008.09.001

    mailto:[email protected]://www.sciencedirect.com/science/journal/01663615http://dx.doi.org/10.1016/j.compind.2008.09.001http://dx.doi.org/10.1016/j.compind.2008.09.001http://www.sciencedirect.com/science/journal/01663615mailto:[email protected]
  • 8/10/2019 PARK - Tangible Augmented Prototyping of Digital Handheld Products

    2/12

    environment can improve the accuracy and tangibility of inter-action with the products. Rapid prototyping is a manufacturingtechnology to generate physical objects so-called RP modelsdirectly from geometric data without traditional tools easily andrapidly [2,14] . An RP model usually serves the purpose of communicating information and demonstrating ideas. It can alsosupport various kinds of tangibility for experiments and interac-tions which gives rapid and critical feedback to the productdevelopment and evaluation.

    The tangible objects, composed of paperand RP models withoutany hardwired connection using electronic components, are easilyavailable at low cost. This makes the AR environment moreaccessible to developers, stakeholders, and even consumers.Moreover, the proposed approach suggests how to combine theforms, functions, and interactions of digital handheld productsphysically and virtually at the same time.

    The rest of the paper is organized as follows: Section 2summarizes previous work related to virtual prototyping. InSection 3, the proposed approach is described with its keycomponents. Section 4 explains the operations of the virtualproduct model in a tangible AR environment. Section 5 addressesthe implementation and application of the product designevaluation system based on the proposed approach. Section 6describes a preliminary user study to show the usefulness of theapproach. Section 7 closes the paper with some concludingremarks and future work to be done.

    2. Previous work

    Early attempts at supporting VP were based on CAD and VR systems. Powerful tools including stereoscopic display systems,head mounted displays (HMD),datagloves andhaptic devices havebeen introduced [9] and combined to construct VP systems thatprovide realistic display of products in a simulated environmentandoffervarious interaction and evaluation means. Bochenek et al.compared the performance of four different VR displays in a designreview setting and mentioned that the best approach for designreview activities could be a combined technology approach [24] .Park et al. suggested virtual prototyping of consumer electronicproducts by embedding HMI functional simulation into VR techniques for design evaluation [15,16] . As it is not easy to builda virtual environment of ne quality and to acquire tangibleinteraction with VR-based systems, many alternative solutionshave been proposed.

    Greenberg and Fitchett presented toolkits called Phigets thatallow designers to explore a tangible user interface (TUI) forinteractive product design [25] . Hartmann et al. presented similartoolkits called d.tools for visually prototyping physical userinterfaces [26] . In TUI, physical objects and ambient spaces areusedto interact withdigital information [27] . Hardwiredconnection

    is often employed using electronic components. Tangible interfacesare quite useful because the physical objects used in them haveproperties and physical constraints that restrict how they can bemanipulated. However, it is difcult to change and evaluate anobjects physical properties dynamically. The human computerinterfaces and interaction metaphors originating from AR researchhave proven advantageous for a variety of applications [17,18] . AR techniques can naturally complement physical objectsby providingan intuitive interface to a three-dimensional information spaceembedded within physical reality. However, although an AR interface provides a natural environment for viewing spatial data,it isoften challengingto interact withandchange thevirtual content.

    To overcome thelimitationsof theAR andTUI approaches whileretaining their benets, tangible AR has been suggested [19,20] .

    Verlinden et al. suggested the concept of augmented prototyping

    that projects the perspective images of the product on the physicalobject made by rapid prototyping techniques [21] . The concept of integrating hardware and software in AR environments has beenpresented [22,23] . Basically, it augments a virtual display onto thesoft mockup of a product by incorporating simple switches as basicinput interfaces. Prototypes with hardwired connection canprovide direct and accurate interfaces, but signicant efforts areusually required to implement and build them. Moreover, it is noteasy to make them available and accessible to many people whoare located at different places.

    Although various ways have been proposed to support virtualprototyping of digital products, more research is still needed in thefollowing aspects. The interaction should be intuitive and tangibleto help developers and users in product design evaluation to makea product of interest more complete and malfunction free beforeproduction. The prototyping environment should be available atlow cost without strong restriction of its accessibility to users.Moreover, foreffective evaluation of theproduct, we need to deneits behavior through forms, functions and interactions, and todevelop a properway of integrating them in a virtual environment.In this paper, we address these aspects by proposing a prototypingapproach called tangible augmented prototyping.

    3. Proposed approach

    Fig. 1 shows the overall process of tangible augmentedprototyping proposed in this paper. There are ve main tasksrequired for relevant prototyping and downstream applications:creation of a product model, acquisition of multimedia contentsdata, generation of HMI functional model, construction of a FSM,and fabrication of AR-based tangible objects.

    Fig. 2 shows a graphical diagram depicting key componentsused for the proposed approach. In the diagram, a game phone isused as an example of a digital handheld product. A product model,multimedia contents data, an HMI functional model, and an FSMconstitute a virtual product model whose operations combinedwith tangible objects in an AR environment facilitate tangibleinteraction, realistic visualization, and functional simulation of theproduct. The visualization of the product in the AR environment isobtained by overlaying the rendered image of the product on thereal world environment in real time [17,18] . For tangible userinteraction, we play with the AR-based tangible objects to createinput events by touching specied regions of the product-typeobject with the pointer-type object. For functional behaviorsimulation, we adopt a state transition methodology to capturethe functional behavior of the product into the HMI functionalmodel, and build the FSM to control the transition between statesof the product using the model. RP and paper-based modeling areproperly adopted to build the AR-based objects which supportgood tangibility for experiments and interactions.

    During the process of tangible augmented prototyping, usersmay detect any problems in the overall appearance, the assemblystructure, or the functional behavior of the product. In such cases,product designers correct the problems and update the productmodel or the HMI functional model. As shown in Fig. 1, the usersand the product designers can promote the product design anddevelopment by repeating the process with the product model andthe HMI functional model updated. In the following subsections,we describe how to acquire the key components used for theproposed tangible augmented prototyping.

    3.1. Product model creation

    Creating a product model is the most basic step forconstructing

    the virtual product model. The product model includes the

    H. Park et al./ Computers in Industry 60 (2009) 114125 115

  • 8/10/2019 PARK - Tangible Augmented Prototyping of Digital Handheld Products

    3/12

    geometry, the attributes of material and color, the assemblystructure, and the kinematics of the part components of theproduct [2] . In general, geometric models of the part componentscan be created with CAD software. In the case that only physicalprototypes or soft mockups are available, the geometric modelscan be created with reverse engineering (RE) tools [28] . In thiswork, we approximate the geometric models by triangular meshesthat are ne enough to provide realistic visualization, and we storethem in OBJ and STL formats [2] . The STL formats are used tofabricate the AR-based tangible objects of the product.

    3.2. Multimedia contents data acquisition

    We acquire multimedia contents data to create visual andauditory display information required for realistic visualization of a product and vivid simulation of its functional behavior. We

    basically require three kinds of multimedia contents data:graphical images, audio sounds, and video animations. Nearlyevery digital handheld product has LCD display(s) to show visualinformation (images and animations) related to its specic states.It also has audio output devices or speakers to output auditoryinformation, that is, audio sounds specic to its states. We use the JPEG le format for graphical images, MP3 or WAV le formats foraudio sounds, and the AVI le format for video animations. Themultimedia contents data can be acquired by audio/videorecording.

    3.3. Generation of HMI functional models

    The HMI functional model of a product is an information modelthat represents the HMI-related functionalbehavior of the product.In this work, we adopt a state transition methodology [26,2931]to capture the functional behavior by breaking it down into thefollowing entities: (1) all objects related to HMI tasks, (2) all HMIevents occurring in the product, (3) all states in which the productcan be, (4) activity information related with each state, (5) statetransitions occurring in each state, and (6) event-condition-action

    information related with each state transition.Nearly every digital handheld product has some part compo-

    nents (i.e. lamps, switches, buttons, sliders, displays, timers, andspeakers) involved in the interaction between the user and theproduct. They are called objects making the basic building blocks of functional simulation. Every object has a pre-dened set of properties and functions that describe everything it can do in areal-time situation. The overall functional behavior of a productcan be broken down into separate units called states . Every statetransition is triggered by one or more events associated with it. Asshown in Fig. 3 , thetransition occursonly when oneof these eventsis activated and some specied conditions are met. Tasks calledactions can be performed before transition to a new state. Taskscalled activities are performed in each state, and they only occur

    when their state becomes active. Actions and activities areFig. 2. Key components of the proposed approach and their relations.

    Fig. 1. Overall process of tangible augmented prototyping.

    H. Park et al./ Computers in Industry 60 (2009) 114125116

  • 8/10/2019 PARK - Tangible Augmented Prototyping of Digital Handheld Products

    4/12

    constructed using the objects properties and functions. Eachaction or activity consists of a set of statements, each of which canbe the assignment of some value to a variable, the calling of afunction of an object, or a composite statement with a conditionalstatement.

    In this work, we have used a simple markup language torepresent the functional behavior of the product [16] . Afteranalyzing and gathering the above-mentioned entities of thefunctional behavior, we write them into a set of text les based onthe markup language to build the HMI functional model. In theHMI functional model, each action or activity consists of a set of logical statements expressed with the objects properties andfunctions. The concrete execution according to each action oractivity is conducted during functional simulation.

    3.4. Construction of a nite state machine

    The FSM is used to control the transition between states of theproduct based on the HMI functional model. In this work, wedevelopeda module to compile the text les forthe HMI functionalmodel and to operate the FSM. Fig. 4 shows the control ow of theFSM module. Note that actions or activities, logically dened in theHMI functional model, are invoked during functional simulation.As they may contain statements invoking functions related toimage synthesis and display, playing audio sounds, and handlingvideo animation data, some function libraries for processing audio/image/video data are often required.

    3.5. Fabrication of AR-based tangible objects

    To improve the accuracy and tangibility of the interactionbetween the user and the product in an AR environment, we usetwo types of AR-based tangible objects: one is for the product, andthe other is for a pointer. The user creates HMI events by touchingspecied regions of the product-type tangible object with thepointer-type tangible object. Each tangible object has at least oneAR marker used to augment the image of the real world with itsrendered image [17,18] .

    For the AR-based tangible object of the product, we build an RPmodel using rapid prototyping with the STL format of the product[2,14] , and paste AR markers on the specied regions of the RPmodel. As nearly every digital handheld product has at least oneLCDdisplay, the AR markers are pasted on the LCD display(s). Fig. 5shows the AR-based tangible object for a game phone. Althoughthe product consists of part components, it is enough to make theRP model as a single component since the necessary informationfor tangible interaction is the position of the pointer-type objectwith respect to the RP model as shown in Fig. 5(b).

    For the AR-basedtangible object of the pointer,we apply paper-based modeling as follows: we generate a polygonal meshcomposed of a cube and a square pyramid, develop its unfoldedsheet,cut outthe sheet,and build thepapermodel with thecut-outsheet. Fig. 6 shows the AR-based tangible object for the pointer.Note that four AR markers are included in the unfolded sheet. Thepaper model actually can be replaced with any physical model (i.e.RP model or plastic object) that satises the following conditions:

    Its shape and size are nearly the same as those of the geometricmodel of the pointer.

    It is easily fabricated at low cost. It is rigid enough to keep its overall shape and the accuracy of picking operations when users grasp it in their hands and touchrigid objects with its tip.

    Paper-based modeling provides low-cost benets with ease tofabricate. Moreover, it can guarantee good rigidity if a paper modelis made of thick and sturdy paper. In this work, the geometricmodel of the pointer is dened as the outward offset of thepolygonal mesh by a small distance. This offsetting is helpful to

    Fig. 3. State transition.

    Fig. 4. Control ow of the nite state machine.

    Fig. 5. AR-based tangible object for a game phone: (a) the RP model with an AR marker and (b) the augmented image of the game phone.

    H. Park et al./ Computers in Industry 60 (2009) 114125 117

  • 8/10/2019 PARK - Tangible Augmented Prototyping of Digital Handheld Products

    5/12

    overlay properly the rendered image of the pointer on the image of the real paper model. The use of the AR-based tangible objects cangreatly improve the accuracy and tangibility of user interaction.

    4. AR-based tangible interaction and functional simulation

    After obtaining the product model, multimedia contents data,the HMI functional model, the nite state machine, and AR-basedtangible objects, we can start the design evaluation of the productby operating them in the AR environment. In this work, ARToolKit[17] is adopted to construct a computer vision-based AR environment. Fig. 7 shows a schematic diagram for AR-basedtangible interaction and functional simulation. When the usermanipulates AR-based tangible objects to touch specied regionsof a digital handheld product, the AR engine recognizes HMI eventsbased on the spatial relations between the AR-based tangible

    objects, reacts to the events, and sends the properresults to outputdevices. It may changestates of theproductand activate associatedactions or activities. Through the output devices, the user canexperience the appearance, kinematics animation, and functionalbehavior of the product.

    4.1. Tangible user interaction

    Most digital handheld products have buttons to push or slidersto move. In order to create HMI events, the user holds the pointer-type tangible object and touches specied regions (for example,buttons or sliders) of the product-type tangible object with the tipof thepointer-type tangible object. We consider a button (orslider)to be pushed or moved (that is, an HMI event occurs) if the

    following conditions are satised:

    The distance from the tip to the button or slider is the shortestamong thedistances from thetip to theotherbuttons andsliders.

    The distance is kept smaller than a tolerance during a speciedtime period.

    According to the length of the specied time period, the HMIevents related to push-typebuttons can be recognized differentlyshort or long pushing. As the distance computation should be

    Fig. 6. AR-based tangible object for a pointer: (a) an unfolded sheet for a paper model; (b) the paper model with four AR markers; (c) the augmented image of the pointer.

    Fig. 7. Control ow in AR-based interaction and simulation.

    H. Park et al./ Computers in Industry 60 (2009) 114125118

  • 8/10/2019 PARK - Tangible Augmented Prototyping of Digital Handheld Products

    6/12

    performed in a reference coordinate frame, it is required totransform the tip, the buttons, and the sliders into the cameracoordinate frame for each video image. Using the cameracalibration information acquired by ARToolKit [17] , we can easilyacquire coordinate transformations between the camera and theAR markers associated with their tangible objects as shown inFig. 8(a). A point p k in a local coordinate frame OXYZ k dened by akth AR marker can be expressed as a point p c in the cameracoordinate frame OXYZ c as follows: pc R

    kc p k d

    kc where the

    rotation matrix R kc and the translation vector dkc constitute a

    transformation from OXYZ k to OXYZ c . The distance d(p 1 , p 2 )between p 1 in OXYZ 1 and p 2 in OXYZ 2 can be easily computed asfollows: dp 1 ; p 2 j jR 1c p 1 d 1c R 2c p 2 d 2c jj, see Fig. 8(b). Toreduce the computational load, we can simply compute thedistances from the center of the tip to the centers of buttons orsliders.

    4.2. Functional behavior simulation

    The functional simulation of the product is completed asfollows: when the user creates an input event with AR-basedtangible objects, the AR engine checks if the event is related to thefunctional behavior of the product or not. If so, the FSM modulerefersto the HMI functional model of theproductand determines if the event triggers a state transition. If the state transition isconrmed,the FSM module does the specied actions and changes

    the state to a new one, and performs the activities of the new state(see Fig. 3). Otherwise, it keeps conducting the activities of thecurrent state. These actions and activities include tasks such as, forexample, changing the position and orientation of buttons andswitches,playing or pausing MP3 music,turning on or offthe lamp,increasing or decreasing the volume. The execution of the actionsand activities yields state-specic visual and auditory data. Thestate-specic visual data are sent to the AR engine to update thevisualimageof theproduct, andthe state-specic auditory data aresent directly to auditory output devices.

    5. Implementation and application

    Based on the proposed approach, a product design evaluation

    system has been implemented in C and C++ languages on a

    windows-based IBM compatible personal computer. As inputdevices, we used two types of AR-based tangible objects and a PCcamera with 640 480 resolution. As output devices, we used anLCD monitor and a pair of speakers. VR-oriented devices such asHMD can also be integrated into the AR environment. Fig. 9 showsthe system environment.

    For product model generation, we used a CAD software calledRhino3D TM version 3.0 and an RE software called RapidForm TM

    version 2004. We stored the HMI functional models of digitalhandheld products into markup language-based text les and usedthem in the FSM module. Graphical images, video animations, andaudio sounds required as multimedia contents data were acquiredby recording. The AR engine is based on ARToolKit [17] andincludes additional modules for visualization, I/O interfacehanding, sound play, LCD image display, and environmentparameter setting. ARToolKit was used for camera calibration,marker recognition, and 3D object augmentation. The LCD imagedisplay module is used to create state-specic images whichappear on the LCD display of each product [15,16] . For thevisualization module, we used OpenGL and GLUT as graphicslibraries. For the sound play module, we used Direct Show for MP3decoding. We developed the other modules by writing our ownsource code.

    Fig. 8. Distance computation in the AR environment: (a) coordinate transformation between a camera and an AR marker; (b) distance between two points using the cameracoordinate frame.

    Fig. 9. System environment for tangible augmented prototyping.

    H. Park et al./ Computers in Industry 60 (2009) 114125 119

  • 8/10/2019 PARK - Tangible Augmented Prototyping of Digital Handheld Products

    7/12

    Figs. 10 and 11 show the virtual product models (i.e. virtualprototypes) of an MP3 player and a game phone in four differentstates and their operations in the tangible augmented prototypingenvironment,respectively. TheMP3 playeris made by iRiver TM andthe game phone is made by LG Electronics TM . In some states, videoanimations are displayed and sounds (i.e. button click or musicsounds) are played. While performing the design evaluation of anyproduct using the system, users may detect any problems in theoverall appearance, the assembly structure (if the prototypingsystem is elaborated further), or the functional behavior of theproduct. In such cases, product designers correct the problems andupdate the product model or the HMI functional model (i.e.feedbacks occur during the iterative process in Fig. 1). The usersand the product designers can promote the product design anddevelopment by repeating the product design evaluation in thismanner.

    6. Preliminary user study

    Usabilitytesting is used forensuringthat theintendedusersof aproduct can carry out the intended tasks efciently, effectively andsatisfactorily. In usability testing, users are asked to performcertain tasks in an effort to measure the products ease-of-use, tasktime, and the users perception of the experience [1,32,33] . Toinvestigate the usefulness and quality of the proposed approach,

    we carried out a preliminary user study of the MP3 player and thegame phone with a subject group consisting of 10 universitystudents. Of the 10, 8 learned the basics of 3D geometric modelingfrom CAD/CAM courses. Simple task performance measures andquestionnaires were used to evaluate the approaches using fourdifferent virtual prototypes: traditional 2D screen prototypes(2DSCR), 3D stereoscopic prototypes (3DSTR), 3D augmentedprototypes (3DAR), and 3D tangible augmented prototypes(3DTAR). As we have built the virtual prototypes from commercialproducts based on reverse engineering, we could include the use of real products (REAL) in the tests as the target reference of theprototyping approaches. Obviously, using a real product is the bestbut most costly approach to its design evaluation.

    As shown in Fig. 12 , 2D screen prototypes present front and/orside views of products, but the image of the views usually has alimited resolution. 2D screen prototypes were built by combiningthe images of views with multimedia contents data, functionalmodels, and FSMs. These 2D screen prototypes allow users toperform the image-based functionalsimulationby clicking buttonson the views. 3DTAR corresponds to the approach proposed in thispaper and 3DSTR corresponds to the VR-based prototypingapproach proposed by Park et al. [15,16] . Similar to the 3DTAR approach, each virtual model in 3DSTR also consists of a productmodel, multimedia contents data, a functionalbehaviormodel, andan FSM.

    Fig. 10. MP3 player in four different states: (a) MP3 Play; (b) Mode Select; (c) FM Radio; (d) Hold; and (e) its tangible augmented prototyping.

    H. Park et al./ Computers in Industry 60 (2009) 114125120

  • 8/10/2019 PARK - Tangible Augmented Prototyping of Digital Handheld Products

    8/12

    Fig. 11. Game phone in four different states: (a) On; (b) Calling; (c) Multimedia menu; (d) Movie; and (e) its tangible augmented prototyping.

    Fig. 12. 2D screen prototypes for (a) the MP3 player and (b) the game phone.

    H. Park et al./ Computers in Industry 60 (2009) 114125 121

  • 8/10/2019 PARK - Tangible Augmented Prototyping of Digital Handheld Products

    9/12

    As shown in Fig. 13, a user using 3DSTR wears HMD andinteracts with thevirtual modelby using some keys anda mouse toexperience the realistic appearance and functional behavior of aproduct. 3DAR is the same as 3DTAR except that, instead of tangible objects madeby RP and paper-based modeling, traditionalAR markers shown in Fig. 14 are used for the augmentation of 3Dvirtual objects and the interaction between users and products.3DAR was included in order to show the advantages that AR-basedtangible objects have over traditional AR markers in aspects suchas tangibility and ease-of-use.

    Note that a 3D screen prototype (3DSCR) working on a singlemonitor screen without an HMDcan be consideredas an alternativeapproach. However, 3DSCR has been compared with 3DSTR in theliterature (for example, see [15,16,32] ), and it was known that3DSCR and3DSTRshow similarresultswith some tradeoffsbetweenthem. As advanced HMDs have been introduced, 3DSTR producesbetter results in many aspects than 3DSCR. Based on this rationale,3DSCR was not included in the user study of the paper.

    In order to evaluate the task performance of the prototypingapproaches, we asked the subjects to complete two tasks for theMP3 player and three tasks for the game phone. Details of the tasksare described in Table 1 . We rst introduced the subjects to fourkinds of prototypes (2DSCR, 3DSTR, 3DAR, 3DTAR) of the twoproducts for 15 min. Each subject was given some time (about20 min) to learn how to manipulate them (i.e. how to click buttons,how to move, scale, and rotate 3D prototypes, and how tomanipulate tangible objects or AR markers). The subject was thenasked to conduct the ve tasks using the four prototypes in thefollowing order: {2DSCR, 3DSTR, 3DAR, 3DTAR} ! REAL. For eachprototype, the tasks were assigned to the subject in the followingorder: T1 ! T2 ! T3 ! {T4, T5}. The prototypes and the tasks inbraces {} were ordered randomly. This random ordering was usedto minimize the learning effect occurred during the repetitive

    tasks. Before performing each task, thesubject could have access toa simple graphical manual describing the steps required tocomplete the task. The results of the performance measures aresummarized in Table 1 and plotted in Fig. 15 .

    After completing all the tasks, each subject was asked to llquestionnaires in order to capture qualitative aspects (i.e. under-standability of functions, ease-of-use, tangibility, sense of realism,legibility) of his or her experience of four prototyping approaches(2DSCR, 3DSTR, 3DAR, 3DTAR). Verlinden et al. and Park et al.performed similar qualitative comparison of their virtual proto-typing approaches [16,33] . The questions asked are summarized inTable 2 . All responses were scored on a ve-point scale and eachquestion included a eld to add some comments. Fig. 16 shows theanalysis results of the questionnaires collected from all thesubjects.

    From the results of task performance, we found that theoverall ranking of task performance was as follows:REAL > 3DSTR > 2DSCR > 3DTAR > 3DAR. We found that the easeof button clicks is the most dominant factor affecting the taskperformance. Button clicks are done with some keystrokes and themouse in 2DSCR and 3DSTR, with simple AR markers in 3DAR, andwith AR-based tangible objects in 3DTAR.Using thekeystrokes andthe mouse was faster and easier than using the AR-based markersor tangible objects. Especially, all the subjects felt severeinconvenience in clicking buttons with traditional AR markers.When they tried to click buttons with simple AR markers, theyoften encountered confusing situations in which 3D virtual objectsoverlapped each other (i.e. the pointer object penetrated theproduct object). This can explain why the 3DAR approach requiredmuch more time to complete the tasks than the others. On theother hand, the subjects mentioned that they did not experienceany object overlap and penetration and felt the sense of touchwhen using tangible objects in the 3DTAR approach.

    Fig. 13. VR-based prototyping of (a) the MP3 player and (b) the game phone.

    Fig. 14. AR-based prototyping: (a) simple AR markers; (b) manipulation of the MP3 player; and (c) manipulation of the game phone.

    H. Park et al./ Computers in Industry 60 (2009) 114125122

  • 8/10/2019 PARK - Tangible Augmented Prototyping of Digital Handheld Products

    10/12

    We also found that the complexity of button layout and thevisibility of prototypes affect the task performance. For the MP3player, it has a few buttons, but their layout in the 2D screenprototype is rather confusing as the buttons were distributed intothree views (one front view and two side views). This often causedthe subjects to make mistakes when picking buttons. For the gamephone, it has over 30 buttons whose layout is rather complex, and

    some buttons aresmalland compact. Moreover,the visibility of the2D screen prototype in either case is not good. Some subjectscommented that they felt inconvenience and made mistakes whenclicking buttons with the mouse pointer in the 2DSCR approach. Inthe 3DSTR approach, subjects can go closely to the prototypes tosee them in ner detail. This helps to make the subjects clickbuttons more easily and accurately. As Kuutti et al. pointed out[32] , it might be unnatural to go more closely to an object thanwhere the eye can be focused. Nonetheless, it must be one of advantages of using 3D virtual prototypes to manipulate (move,scale,and rotate) freely them in a VR environment. In the AR-basedapproaches, subjects can also have a close look at the prototypes aslong as theAR markers associated with theprototypesare capturedand identied. The visibility of prototypes in the AR-basedapproaches is not better than the one in the 3DSTR approachsince the resolution of PC camera is lower than that of HMD. On theother hand, some subjects without experience of using HMDcommented that in the 3DSTR approach they had some incon-venience in visibility of 3D prototypes and felt some unnaturalweight gain in their heads while wearing the HMD.

    From the analysis results of the questionnaires, we found theadvantages of 3D-based approaches over the 2D-based approachin most aspects, the signicant advantages of 3DTAR over 3DAR,and some trade offs between 3DSTR and 3DTAR. As the same HMIfunctional models were integrated into all the four kinds of prototypes, there were no signicant differences between the

    four approaches in understandability of the product functions.The 3D-based (3DSTR, 3DAR, 3DTAR) approaches gave thesubjects better sense of realism than the 2DSCR approach.

    Table 1Task descriptions and performance measures.

    Product type Task Task steps Average time required (in s)

    2D SCR 3D STR 3D TAR 3D AR REAL

    MP3 Player (T1) Play the 3rd MP3 musicwith volume level 30.

    1. Turn on the MP3 player. 6.5 5.9 9.2 42.4 5.32. Play the music.3. Move to the 3rd music.

    4. Increase the volume up to 30 levels.5. Hold on the buttons (move hold slider to the right).(T2) Play the 2nd FM radiostation with volume level 20.

    1. Hold off the buttons (move hold slider to the left). 8.8 8.1 13.0 45.8 7.52. Enter the menu.3. Select the FM radio submenu to play FM radio.4. Move to the 2nd FM radio station.5. Decrease the volume down to 20 levels.

    Game Phone (T3) Make a call to the1st tester.

    1. Turn on the game phone. 10.0 7.7 13.3 47.8 6.42. Enter the phone number by using number buttons.3. Press call button.

    (T4) Play an animation. 1. Enter the main menu. 9.6 7.3 10.3 42.4 6.72. Move to the multimedia submenu (press 6 button).3. Select the 2nd bin for animation les.4. Play the animation by pressing OK button.

    (T5) Change the callerring into the 3rd one.

    1. Enter the main menu. 9.0 6.6 8.8 35.2 5.62. Move to the caller ring submenu (press 2 button).3. Select the 3rd one.4. Set it to the current caller ring by pressing OK button.

    Fig. 15. Graphical plots of task performance measures of (a) AR-based approaches

    and (b) four approaches.

    Table 2Questionnaires contents (translated from Korean).

    Q1 Can you understand the product functions by using the virtual prototype?Q2 Is it easy to click buttons when using the virtual prototype?Q3 Does the virtual prototype make you feel as if you push real buttons?Q4 Does the virtual prototype look like the real product?Q5 Can you gure out the size of the product with the virtual prototype?Q6 Can you feel a three-dimensional effect when using the virtual prototype?Q7 Is the liquid crystal display (LCD) of the product legible?Q8 Do you think the use of virtual prototype interesting?Q9 Does the virtual prototype offer enough information for a decision to buy the

    product?

    H. Park et al./ Computers in Industry 60 (2009) 114125 123

  • 8/10/2019 PARK - Tangible Augmented Prototyping of Digital Handheld Products

    11/12

    Regarding the three-dimensional effect, the subjects perceiveddepth the most vividly in the 3DSTR approach. As theseapproaches allow the subjects to go closely to the prototypesand to see the display of the products more clearly, they gothigher points than the 2DSCR approach in legibility. The 3DAR approach received thelowest points in theease of button clicks asmentioned above. The 3DAR and 3DTAR approaches showedsimilar results except that the 3DTAR approach received muchhigher points than the 3DAR approach especially in tangibilityand the ease of button clicks. The 3DSTR and 3DTAR approachesreceived higher points than the 2DSCR approach in most aspects,but we found some tradeoffs between the two: 3DTAR receivedlower points than 3DSTR in the ease of button clicks but higherpoints in the tangibility and the sense of realism. Some of thesubjects felt some difculty in guring out the size of the productwith the 2DSCR and 3DSTR approaches. However, all the subjectscould easily estimate the product size in the AR-basedapproaches since they felt like holding the product with theirhands (especially in 3DTAR). Most of the subjects answered that3D-based prototypes were more appealing to customers than the2D screen prototype.

    Some subjects pointedoutthattheysometimesfelt confusionanddifculty in their tasks when the virtual objects disappeared in theAR-based approaches. This problem usually occurs when some AR markers fail to be recognized due to bad light conditions or someocclusions between the markers and by the users hands. Somesubjects expressed inconvenience as they had to watch the LCDscreen (not their hands) while manipulating tangible objects or AR markers.Nonetheless,mostofthesubjectsfeltthatitwasverynicetotouch and grasp the prototypes in the 3DTAR approach. Somesubjects commented that it would be better if they could clickbuttons with their ngers not with the pointer-type tangible object.Theyalsoaddedthatthe3D-basedprototypes,iftheycanbeaccessed

    via Internet, will draw great interest from a number of remotecustomers. Based on the preliminary user study, we found that thesubjects feedback about the3DTARapproachwas encouragingsinceit could provide them with more tangibility and better sense of realism while allowing them to experience the functional behaviorand the visual appearance of the product easily and vividly.

    7. Concluding remarks and future work

    Functions represent what a product does to satisfy customers.These functions usually differ from product to product. Generally,there is no prototyping tool that canmake a prototype representallthe functions of every kind of products. It is common to use variousprototypes each of which can represent specic functions of a

    group of target products.

    In this paper, we have proposed a novel approach to virtualprototyping of digital handheld products, which is called tangibleaugmented prototyping. The primary function of these products ismostly considered as sending users visual or auditory informationin response to user inputs. The proposed approach is aimed atgenerating and utilizing prototypes that can represent not only theprimary function but also other functions such as looking nice inaesthetic shape and keeping good in overall structure. In theapproach, a product model, multimedia contents data, an HMIfunctional model, and an FSM are combined with AR-basedtangible objects whose operations in an AR environment facilitatethe tangible interaction, realistic visualization, and functionalbehavior simulation of a digital handheld product. We presentedhow to adopt rapid prototyping and paper-based modelingproperly in building the AR-based tangible objects, and therebyto realize accurate and tangible interaction between the user andthe product. We also suggested how to combine the forms,functions, and interactions of digital handheld products physicallyand virtually at the same time.

    The AR-based tangible objects are composed of paper and RPmodels without any hardwired connection. If drawing les forpaper models and STL les for RP models are sent (via Internet),anyone can easily obtain the tangible objects by simple papercrafting and with the help of an RP service bureau. The AR environment described in this paper is easy to implement,available at low cost, and accessible to developers, stakeholders,and even consumers.

    The paper model used as a pointing tool can be replaced by ahaptic device in the AR environment. With the haptic device, wecan enhance the sense of touch and improve theaccuracy of buttonclicks during design evaluation. However, using the haptic devicemakes object manipulation rather uncomfortable due to increasedspatial constraints. It also increases hardware costs directly, whichsignicantly restricts the availability of the AR environment tousers.

    Based on theproposed approach, a prototyping systemhas beendeveloped and applied for the design evaluation of various digitalproducts such as MP3 players and mobile phones, and it hasobtained highly encouraging feedback from users. We found somepotential possibility that the prototyping system can be used as animportanttool for design review and evaluationof digital handheldproducts. We also found that the proposed approach can beapplicable to any product (or system) that satises the followingguidelines:

    The primary function of the product is featured as sending usersvisual or auditory information in response to user inputs.

    Users can create HMI inputs by touching, moving, or pressing thespecic components of the product.

    The product has a at region in which an AR marker can be

    placed. The tangible physical model (i.e. RP model) of the product can beeasily available at low cost.

    These guidelines arenot so tight that theapproachwouldnot beapplicable to various kinds of products including digital handheldproducts. We are currently expanding the application of theapproach to various products (or even systems) used forentertainment, education, and training.

    From the user study, the prototyping system revealed somepoints which guide the directions of our future research forimproving the proposed approach. Firstly we will improvecompatibility between the manipulation and the viewing of tangible objects. In our present AR environment, the direction

    from the users eyes to the LCD screen is different from the

    Fig. 16. Graphical plots of questionnaire results for all subjects.

    H. Park et al./ Computers in Industry 60 (2009) 114125124

  • 8/10/2019 PARK - Tangible Augmented Prototyping of Digital Handheld Products

    12/12

    direction from the camera to an object of interest. This tends tomake the users cognitive process inconvenient. We expect toalleviate this problem greatly by adopting an optical or video see-through HMD system in the AR environment. Secondly we willmake the marker recognition module more robust to reduce theoccasions where the virtual objects disappeared during theiraugmentation. Thirdly we will incorporate computer visiontechniques to recover the image of real objects (e.g. ngers)occluded by the image of virtual objects and thereby to make thevisualization more natural and realistic. Fourthly we will makeuser interaction more tangible by devising a picking mechanismwith which the user can push or select buttons with his or herngertips. Lastly we will expand the system to be run on web-based environments that allow easy access to AR-based productdesign evaluation via Internet.

    Acknowledgements

    The authors are grateful to all the anonymous referees for theirhelpful comments on this paper. This work was supported by theKorea Research Foundation Grant (KRF-2008-013-D00152).

    References

    [1] K.T. Ulrich, S.D. Eppinger, Product Design and Development, McGraw Hill, NewYork, 2004.

    [2] K. Lee, Principles of CAD/CAM/CAE Systems, Addison Wesley, Berkeley, 1999.[3] M.J. Clayton, J.C. Kunz, M.A. Fischer, Rapid conceptual design evaluation using a

    virtual product model, Engineering Applications of Articial Intelligence 9 (4)(1996) 439451.

    [4] H.J. Bullinger, R. Breining, W. Baucer, Virtual prototypingstate of the art inproduct design, in: Proceedings of the 26th International Conference on Compu-ters & Industrial Engineering, 1999, pp. 103107.

    [5] S. Ottosson, Virtual reality in the product development process, Journal of Engineering Design 13 (2) (2002) 159172.

    [6] F. Zorriassantine, C. Wykes, R. Parkin, N. Gindy, A survey of virtual prototypingtechniques for mechanical product development, Journal of Engineering Manu-facture 217 (2003) 513530.

    [7] F. Dai, W. Felger, T. Fruhauf, Virtual prototyping examples for automotive indus-tries, in: Proceedings of Virtual Reality World, 1996, pp. 113.

    [8] M. Soderman, Virtual reality in product evaluations with potential customers: anexploratory study comparing virtual reality with conventional product represen-tations, Journal of Engineering Design 16 (3) (2005) 311328.

    [9] G.C. Burdea, P. Coiffet, Virtual Reality Technology, John Wiley & Sons, USA, 2003.[10] S. Jayaram, H.I. Connacher, K.W. Lyons, Virtual assembly using virtual reality

    techniques, Computer-Aided Design 29 (8) (1997) 575584.[11] N. Shyamsundar, R. Gadh, Collaborativevirtual prototyping of product assemblies

    over the Internet, Computer-Aided Design 34 (2002) 755768.[12] Z. Siddique, D.W. Rosen, A virtual prototyping approach to product disassembly

    reasoning, Computer-Aided Design 29 (12) (1997) 847860.[13] S.K. Ong, L. Jiang, A.Y.C. Nee, An internet-based virtual CNC milling system,

    International Journal of Advanced Manufacturing Technology 20 (1) (2002)2030.

    [14] S.H. Choi, A.M.M. Chan, A virtual prototyping system for rapid product develop-ment, Computer-Aided Design 36 (2004) 401412.

    [15] H. Park,C.Y. Bae, K.H.Lee, Virtual prototyping of consumerelectronic products byembedding HMI functional simulation into VR techniques, Transactions of theSociety of CAD/CAM Engineers 12 (2007) 8794.

    [16] H. Park, J.S. Son, K.H. Lee, Design evaluation of digital consumer products usingVR-based functional behavior simulation, Journal of Engineering Design 19(2008) 359375.

    [17] ARToolKit, http://www.hitl.washington.edu/ARToolKit .[18] R.T. Azuma, A survey of augmented reality, Presence: Teleoperators and Virtual

    Environments 6 (1997) 355385.[19] H. Kato, M. Billinghurst, I. Poupyrev, K. Imamoto, K. Tachibana, Virtual object

    manipulationon a table-top AR environment, in: Proceedings of the InternationalSymposium on Augmented Reality, 2000, pp. 111119.

    [20] M. Billinghurt,H. Kato,I. Poupyrev,Collaboration withtangibleaugmented realityinterfaces, in: Proceedings of HCI International, 2001, pp. 234241.

    [21] J. Verlinden, A. de Smit, A.W.J. Peeters, M.H. van Gelderen, Development of aexible augmented prototyping system, Journal of WSCG 11 (2003) 496503.

    [22] T.J. Nam, Sketch-based rapid prototyping platform for hardwaresoftware inte-grated interactive products, in: Proceedings of conference on human factors incomputing systems (CHI), 2005, pp. 16891692.

    [23] W. Lee, J. Park, Augmented foam: touchable and graspable augmented reality forproduct design simulation, Bulletin of Japanese Society for the Design Science 52(2006) 1726.

    [24] G.M. Bochenek, J.M. Ragusa, L.C. Malone, Integrating virtual 3-D display systemsinto product design reviews: some insights from empirical testing, International Journal Technology Management 21 (2001) 340352.

    [25] S. Greenberg, C. Fitchett, Phidgets: easy development of physical interfaces

    through physical widgets, in: Proceedings of the ACM UIST, 2001, pp. 209218.[26] B. Hartmann, S.R. Klemmer, M. Bernstein, N. Mehta, d.tools: visually prototyping

    physical UIs through statecharts, in: Proceedings of ACM Symposium on UserInterface Software and Technology (UIST), 2005.

    [27] H. Ishii, B. Ullmer, Tangiblebits: towardsseamlessinterfacesbetweenpeople,bitsand atoms, in: Proceedings of Conference of Human Factors in ComputingSystems, 1997, pp. 234241.

    [28] T. Varady, R. Martin, J. Cox, Reverse engineering of geometric modelsan intro-duction, Computer-Aided Design 29 (1997) 255268.

    [29] J. Rumbaugh,State treesas structured nite statemachinesfor userinterfaces, in:Proceedings of ACM SIGGRAPH Symposium on User Interface Software, 1988, pp.1529.

    [30] D. Harel, Statecharts: a visual formalism for complex systems, Science of Com-puter Programming 8 (1987) 231274.

    [31] H. Feng, Dcharts, a formalism for modeling and simulation-based design of reactive software systems, Master Thesis, McGill University, 2004.

    [32] K. Kuutti,K. Battarbee,S. Sade, T.Mattelmaki, T. Keinonen, T.Teirikko, A. Tornberg,Virtual prototypes in usability testing, in: Proceedings of the 34th Hawaii Inter-national Conference on System Sciences, 2001, pp. 17.

    [33] J. Verlinden, W. van den Esker, L. Wind, I. Horvath, Qualitative comparison of virtual and augmented prototyping of handheld products, in: Proceedings of International Design Conference, 2004, pp. 533538.

    Hyungjun Park is an associate professor at theDepartment of Industrial Engineering, Chosun Univer-sity, Korea. He received his BS, MS, and PhD degrees inIndustrial Engineering from Pohang University of Science and Technology (POSTECH), Korea, in 1991,1993, and 1996, respectively. From 1996 to 2001, heworked as a senior researcher at Samsung Electronics,Korea. He involvedin developing commercial CAD/CAMsoftware and in-house software for modeling andmanufacturing aspheric lenses used in various opticalproducts. Since 2001, he has been a faculty member of Chosun University. His current research interestsinclude geometric modeling, virtual prototyping of

    engineered products, 3D shape reconstruction using reverse engineering, bio-medical engineering, and CAD/CAM/CG applications.

    Hee-Cheol Moon received his BS and MS degrees inIndustrial Engineering from Chosun University, Korea,in 2005 and 2007, respectively. He is currently a PhDstudent at Chosun University. His mainresearch topicisvirtual prototyping of portable electronic productsusing augmented reality and CAD/CAM techniques.

    Jae Yeol Lee is an associate professor at theDepartmentof Industrial Engineering, Chonnam National Univer-sity, Korea. Before joining the faculty members of Chonnam National University, he was a seniorresearcher at Distributed Collaboration TechnologyResearch Team in Electronics and TelecommunicationsResearch Institute (ETRI). He received his BS, MS andPhD degrees in Industrial Engineering from PohangUniversity of Science and Technology (POSTECH),Korea, in 1992, 1994, and 1998, respectively. Hiscurrent research interests include collaborative virtualengineering, collaborative product commerce, anddistributed computing for product development.

    H. Park et al./ Computers in Industry 60 (2009) 114125 125

    http://www.hitl.washington.edu/ARToolKithttp://www.hitl.washington.edu/ARToolKit