Automation in Construction 20 (2011) 338–352

Contents lists available at ScienceDirect

Automation in Construction

j ourna l homepage: www.e lsev ie r.com/ locate /autcon

Augmented reality-based computational fieldwork support for equipment operationsand maintenance

Sanghoon Lee a,⁎, Ömer Akin b

a 104 Engineering Unit A, Architectural Engineering Department, The Pennsylvania State University, University Park, PA 16802 USAb 5000 Forbes Ave., School of Architecture, Carnegie Mellon University, Pittsburgh, PA 15213 USA

⁎ Corresponding author. Tel.: +1 814 863 2080; fax:E-mail address: Sanghoon2009@gmail.com (S. Lee).

0926-5805/$ – see front matter © 2010 Elsevier B.V. Adoi:10.1016/j.autcon.2010.11.004

a b s t r a c t

a r t i c l e i n f o

Article history:Accepted 4 November 2010Available online 3 December 2010

Keywords:Operations and maintenanceFieldwork efficiencyAugmented realityBuilding information modeling

This paper describes the development of an Augmented Reality (AR)-based equipment Operations andMaintenance (O&M) fieldwork support application to improve efficiency at the site and the experimentalevaluation of the application. The application consists of an AR-based interface for intuitive GUI and an O&Minformation model developed with various types of O&M specific information obtained through observingtradespeople's fieldwork activities, and those from ComputerizedMaintenance Management System (CMMS)and Building Automation System (BAS). In addition, the BACnet protocol is used to get sensor-derivedoperation data in real time from BAS. A series of experiments was conducted to quantitatively measureimprovement in equipment O&M fieldwork efficiency by using a software prototype of the application. Theresults show that with the application the subjects saved, on average, 51% of time spent at task when theylocated target areas and 8% of time at task while obtaining sensor-based performance data from BAS.

+1 814 863 4789.

ll rights reserved.

© 2010 Elsevier B.V. All rights reserved.

1. Introduction

Operations and Maintenance (O&M) encompasses the activitiesthat Facilities Management Services (FMS) personnel perform toensure that facilities continue to fulfill their intended functions. Morespecifically, operation includes activities performed to providecomfortable working and living environments, whereas maintenanceprovides equipment upkeep to prevent functional failure [1–4]. As theO&M phase is the longest period in the lifecycle of a building, themajority of expenses are naturally accrued during the O&M phase.According to Teicholz [5], more than 85% of total costs spent over thelifecycle of a building are on O&M.

As computing technologies have been developed to support O&M,many traditional methods of operating and maintaining equipmentand facilities have become automated, enabling O&M to become fasterand more reliable. In addition to these advantages, computerizedsystems open enormous research potential in O&M for furtherimprovement. Major companies in the field have been developingand providing computer-aided O&M systems and tools; however,there is further potential improvement in O&M fieldwork from theperspective of information support. For example, if it were possible forMaximo Asset Management System, a commercial maintenancemanagement system, and ARCHIBUS/FM™, another FM supportsystem, to make their data available to each other, O&M personnel

could see their work orders related to one specific area in a graphicalinterface.

While opportunities for research abound, there is a peculiar lack ofprogress in the area of computer-aided O&M. This is due to variousobstacles. For example, the majority of tradespeople are relativelyconservative when it comes to computing environments. Even thoughthey use automated applications such as Computerized MaintenanceManagement Systems (CMMS), those systems are often black boxes tothem. Paper is still the primary medium for transfer of informationbetween tradespeople and computational systems, which can cause atime delay in responding to service requests, resulting in inefficientO&M. As such, in this research, a computational support system isdeveloped in order to reduce the inefficiency factors due to difficultieswith accessing information and data in O&M support systems, andtested to show potential improvement in equipment O&M fieldworkefficiency with the computational support system. We focus onbuilding HVAC systems, especially three-dimensional elements suchas pipes, ducts and valves.

2. Background

The introduction of computing technology in the O&M domain haschangedmany aspects of O&M, including automation of processes andhandling massive data. Consequently, it has raised many researchquestions on how to improve the automated environment for moreeffective and efficient O&M. This section summarizes current trends inpractice and research directions, which focus on the perspective ofcomputing technologies and their impact on O&M.

339S. Lee, Ö. Akin / Automation in Construction 20 (2011) 338–352

2.1. Operation and Maintenance in practice

Although there are many different O&M activities [6–8], themajority of maintenance work (86%) is still in Reactive and PreventiveMaintenance programs. In Reactive Maintenance program, facilitiesand equipment are operated until they fail. FMS personnel repairequipment when clients request maintenance services. Studies asrecent as the winter of 2000 show that more than 55% of maintenanceprograms in the United States are of the Reactive Maintenance type[6]. In Preventive Maintenance program, facilities and equipment aremaintained based on a pre-planned maintenance schedule. 31% fallinto this maintenance type.

Lee and Akin [7] conducted a shadowing experiment in order tomeasure inefficiency in the current O&M fieldwork and to identifybottleneck of the inefficiency. The results of the experiment revealthat the fieldwork inefficiency can be improved up to 20% byproviding proper information support. They also state that eventhough the FMS department studied for the experiment has O&M datain electronic format, the tradespeople preferred hardcopy formatssuch as blue prints and manual books. Therefore the tradespeoplespent extra time on transit and finding documents, which arerepresentative non-value-adding activities. Consequently, there is aneed for the development of a system that provide O&M fieldworkerswith various O&M information related to the specific equipment andthe user's location to facilitate their fieldwork.

2.2. Current O&M information support

There are many perspectives from which to describe current O&Mtrends in practice. The main focus in this research is on computationalsupport to facilitate fieldwork. The noticeable difference fromcontemporary maintenance management systems is that sincecomputerized systems were not popularly used for O&M decadesago, all data such as work requests and resources were exchangedbetween O&M personnel and stored in FMS filing systems in hardcopyformat. However, computational technologies enable O&M personnelto reduce or eliminate wasted human effort by providing betterinterfaces to access data that they need, and better tools to save timeand to complete their jobs more safely.

In the 1980s, when Personal Computers (PC) became popular andbegan to dominate the computing environment, applications for O&Mwere also migrated to run on PC. However, because of the change,applications that had been integrated in a single mainframe beforePCs were used popularly became fragmented. Each application ran ona separate PC and was managed locally by each department. Themodel introduced by Rondeau et al. [9] explains the phenomenon ofdata fragmentation.

Since the 1990s, there have been efforts at integration betweenvarious graphic and non-graphic applications [10]. For example,ARCHIBUS/FM™, one of the industry's leading software packages, hasstated that it is compatible with Microsoft Office Products andAutoCAD®, which means Microsoft Office Excel andWord documentscan be linked to ARCHIBUS/FM™'s graphical information. As a result,users can see location-specific information. However, even thoughthis software package provides user-friendly GUI (Graphical UserInterface), it lacks functions that would be critical to maintainingthree-dimensional equipment such as pipes and conduit. Morespecifically, it cannot display 3D geometry on its display interfacedue to its 2D-based interface. Users see a 2D floor plan of a buildingand retrieve, for example, statistical data and reports, but cannot seewalls, pipes passing behind the walls, nor information related to thewalls and pipes. GEMnet (GSA Energy and Maintenance Network) isanother similar example of the integration efforts. It consists ofWebCTRL® (a web-based GUI for building automation systems),PACRAT (Performance And Continuous Re-commissioning AnalysisTool), and Maximo® (a computerized maintenance management

system) in order to provide fieldworkers with O&M requestsintegrated with equipment performance data.

2.3. Research on O&M information support

A survey conducted by Liu et al. [11] reveals that overall around50% of organizations did not have O&M information ranked as “veryhelpful” or “extremely helpful” (the second and the highest ranks) inthe survey. More significantly, only 12% of the respondents said thatthe condition of the equipment/facility existed in their systems; 15%had maintenance records; 34% had information on installation date;and only 36% had responsible division/person information in theirdatabases. In addition, the research team stated the necessity ofdeveloping properly coordinated methods and structures to collect,store, and retrieve as-built information. Four years later, Clayton andhis research group [12] surveyed USSA (United Services AutomobileAssociation) facilities management personnel on satisfaction andimportance levels of O&M information. The survey results reveal thatmore than half of the O&M information items received a 3 or 4satisfaction rating on a scale of 5, which means that the respondentswere relatively satisfied with the O&M information items in theirsystems. However, the results also reveal that Valve tag schedule,Electrical breaker panel schedule, and Mechanical design rationalereceived the lowest satisfaction ratings, whereas they were all ratedhigh in terms of importance.

There have been efforts to standardize information used forseamless information exchange among AEC/FM participants. IFC(Industry Foundation Classes) is one of the leading BuildingInformation Models (BIM) developed for the AEC/FM industry. Itintends to support the seamless exchange of building informationamong model-based applications from various perspectives. Forexample, the IFC Model-Based Operation and Maintenance ofBuildings (Ifc-mBomb) project [13] was conducted to support smoothdata flow from the design and construction phases to the O&M phaseusing IFC models. In addition, Arnold and his research team [14]showed successful data transfer between applications, not only in thesame field, but also in different fields.

Location-specific computing technologies allow users to automat-ically retrieve information only associated with the specific facilityand equipment. Hammad et al. [15] developed a prototype of LBC-Infra (Location-Based Computing for Infrastructure) and conducted acase study on a bridge in Pittsburgh, PA. The LBC-Infra is intended tosupport bridge inspectors during fieldwork by providing informationat any given location and orientation. The prototype integrates spatialdatabases, mobile computing technology, tracking technologies, andwireless communications. The feasibility of the LBC-Infra wasinvestigated by conducting a case study with the Virtual Realitymodel of a bridge. Although the prototype was not tested with a realbridge environment, it certainly showed the possibility of improvingefficiency in fieldwork and the safety of fieldworkers. Goose et al. [16]developed a computer vision-based location tracking and 3Dvisualization tool combined with speech interaction technology,called PARIS (PDA-based Augmented Reality Integrating Speech).The tool uses a camera and visual markers to identify different piecesof equipment. The camera continuously captures images, and the toolanalyzes each image frame to detect pre-defined markers. Once thetool detects markers, the user is localized and can see location-specificinformation relevant to the equipment at that location using a multi-modal interface.

3. Approach

3.1. Motivation

There is significant potential for improvement in the efficiencyof Operations and Maintenance (O&M) fieldwork. Computer

340 S. Lee, Ö. Akin / Automation in Construction 20 (2011) 338–352

technologies are used in diverse areas throughout the lifecycle of abuilding. However, each system still provides only a part of theinformation needed to perform O&M fieldwork. For example, theGEMnet combines only work orders and performance data. It does notprovide three-dimensional (3D) CAD drawings of equipment andfacilities. Further, PARIS provides an interface to get only performancedata with 3D VRML models. All these types of information are in factnecessary for O&M fieldwork. Furthermore, standardization ofinformation is important in order for different O&M support systemsto exchange various types of information. Although the Ifc-mBombproject demonstrated that the data generated in the design,engineering and construction phases were used in the O&M phase,the project did not include information such as work orders generatedduring the O&M phase.

In order to enhance the existing computational O&M environment,which fulfills only a portion of all functions needed to complete O&Mfieldwork activities, there is a research need for the development of aproper computational O&M fieldwork support system that integratesvarious types of O&M information-building geometries and specifica-tions, and operation and maintenance data and provide them in anintuitive Graphical User Interface (GUI) in the field.

3.2. Research objectives and steps

The objective of this research is to improve efficiency in O&Mfieldwork in terms of time spent and steps taken to complete workorders. This is achieved by providing O&M personnel with relevantdata and information as needed through a software application thatinteracts with databases to retrieve maintenance information andequipment data such as geometric models and specifications andwithBAS for sensor-based performance data in real time.

In order to achieve the research goal, we collect O&M datainstances by observing tradespeople's field activities and categorizethem to develop an O&M information model for the development ofan O&M fieldwork support application. Then, we elicit functionalrequirements based on the results of the field observation; weinvestigate information technologies to find the suitable solutions forthe requirements in terms of an interface and a controller forcomputational O&M fieldwork support. Finally, we develop aprototype application based on the technologies, conduct verificationtests to see if the application can meet the requirements, and evaluatethe performance of the prototype application with actual O&Mpersonnel using representative O&M scenarios to see the improve-ment of O&M fieldwork efficiency.

3.3. Research scope

The scope of this research includes target equipment, target tasksand primary actors. Target equipment refers to three-dimensionalelements of HVAC systems such as pipes, ducts and conduit as well asrelated equipment. These were chosen because they are maintainedby in-house FMS personnel, unlike other equipment, which requiresspecific knowledge and must be maintained by third-party suppliersunder contract. The target equipment is essential because it serves alloccupants, not just in a particular building, but across all buildingswithin a larger complex. Further, some pieces of the target equipmentare hidden and some are exposed, which leads to difficulty inmaintenance, especially in locating exact parts for maintenance. It isalso hard to provide geometric information for the target equipmentusing traditional CAD-based information support systems in the field.Retrieving information from CAD software packages is not an easytask for ordinary tradespeople, who normally prefer blueprints. Thetarget types of O&M fieldwork in this research cover daily services,scheduled and unscheduled urgent priority maintenance, andpreventive maintenance. Among the various activities performedduring fieldwork, activities of primary interest are those related to

O&M information, such as methods to access various types of O&Minformation. The primary actors of this research are O&M fieldworkerssuch as FMS tradespeople, and the stakeholders include O&Mmanagers, operation engineers and foremen. The O&M data typesused for this research are obtained through a field observation,Maximo® system and object properties defined in BACnet.

4. The development of an AR-based O&M Fieldwork Facilitator

This section describes the development of the Augmented Reality(AR)-based Operation and Maintenance Fieldwork Facilitator(AROMA-FF). We elicit functional requirements by observingfieldwork activities that actual O&M fieldworkers perform at thesite. We conducted and investigate different technologies to find asuitable solution to fulfill the functional requirements. Then, wedevelop the architecture of AROMA-FF and its prototype application.Finally we test the prototype application with pieces of real HVACequipment and facilities they serve to see if it fulfills the functionalrequirements.

4.1. Functional requirements

Fig. 1 shows three primary functions in an IDEF0 (IntegratedDEFinition method) diagram [17]: identify equipment and facilities,retrieve and visualize O&M information (maintenance-related, oper-ation-related and geometric representation).

In order to facilitate O&M fieldwork, many technologies have beendeveloped that could potentially automate the identification ofequipment and facilities as well as the acquisition of information atthe site. For example, RFID (Radio Frequency Identification) and GPS(Global Positioning System)-based tracking systems have the capacityto automatically identify equipment and facilities based on user'slocation. Web browsers and PDF document format have been used tointegrate different types of information (i.e., text, drawings, andgraphs) in a unified display interface on a portable tool. Together,these approaches could eliminate extra time for the purpose of gettinginformation. However, those user-tracking systems have limitationsof accuracy when users are inside complex buildings. Also, 2D-basedGUI and documents pose a challenge to fieldworkers: they mentallytransform the two-dimensional drawings and diagrams on hisportable device into the three-dimensional environment of hisworksite. While many experienced fieldworkers can do this withease, they must still rely on their own intuition and interpretation topinpoint the exact location of equipment, which can lead to mistakesand wasted time.

Augmented Reality (AR) technology fulfills the functional require-ments of identifying equipment and facilities as well as integratingvarious types of information at the site. More importantly, ARtechnology resolves the challenge of interpreting two-dimensionaldrawings and diagrams by superimposing geometric representationof equipment and/or facilities onto real objects so that the user can seethem immediately and accurately. For this reason, AR technology isone of the most suitable interfaces for O&M fieldwork support.Therefore, using the AR technology, AROMA-FF is designed to takevisual markers to identify pieces of equipment and facilities, or directuser-input about equipment/facility. A computer vision-based objectidentification module is used to detect visual markers in a videostream. The visualization function accesses databases to retrieve anyequipment-specific information. The main outputs are integratedequipment/facility-specific data and information in a unified userinterface. Then, the retrieved data are superimposed onto a videostream supplied through the camera installed in AROMA-FF. With thisdisplay method, the user would be able to see a hidden portion of theequipment or facility in full scale.

Fig. 1. Four primary functions in IDEF0 diagram format.

341S. Lee, Ö. Akin / Automation in Construction 20 (2011) 338–352

4.2. Operations and Maintenance information

In order to effectively provide FMS personnel with various types ofO&M information at the site, an O&M informationmodel is developed.Information instances are collected and compared to find a set ofinformation types commonly used for O&M fieldwork. The informa-tion types are categorized into the following three groups: mainte-nance information, operation data and BIM-based equipment/facilityinformation. The sources for the data collection include ComputerizedMaintenance Management Systems (CMMS) and Building Automa-tion Systems that are used by the institute studied in this research aswell as direct collection by observing tradespeople's fieldwork at thesite.

4.2.1. Identification of O&M information types from the field observationThe total number of 618 information instances was obtained by

observing 58 O&M work orders that actual O&M fieldworkers tookcare of. The information instances are compared using the Compar-ative Analysis Table [18] to come up with a set of representativeinformation types. The total number of 20 information types isidentified. In the pre-fieldwork phase, fieldworkers collect funda-mental O&M data such as requestor's name and contact information,location and problem description. Information instances used duringfieldwork at the site are mostly related to parts and materials neededto complete the task as well as contact information for otherpersonnel, if necessary. When a work order is related to theperformance of equipment, tradespeople need performance-relateddata for equipment diagnosis such as current temperature set pointsand actual temperature.

4.2.2. O&M information used through CMMS and BASIn addition to fundamental maintenance data types such as

equipment specifications and drawings of equipment and facilities,we further collected maintenance information types through theMaximo® asset management system that is used by the FMSdepartment studied in this research to manage maintenanceinformation, including information about FMS personnel, sub-con-tractors and equipment, as well as work orders. The types ofequipment information managed in the system are different fromthose provided by themanufacturer such as equipment specifications.They are collections of information types that are specialized tomaintaining equipment rather than those that describe the equip-ment itself. Examples of the former information type includeequipment IDs that are assigned by the owner organization andvendor and manufacturer names as contact information, whereasthose for the latter would include the maximum capacity of the

equipment and the resolution of a sensor. Consequently, the majorityof information types in CMMS are common attributes that can be usedregardless of equipment types.

The types of operations information include operation logics,facilities conditions, and live data streams from deployed sensors, setpoints and control parameters, alarms and events, and trend logs. In thisresearch, we focused on set-point data and sensor-derived performancedata to indicate the status of equipment and facilities that theequipment serves because checking these types of data are the firstcore activity to diagnose the current status of equipment. By checkingset points and sensor-driven performance data to see if the equipmentresponds accordingly, they can make correct decisions of what shouldbe done to fix the problem. In order to collect those data, we took a lookat BACnet (Building Automation and Control Networks), which arecommonly used as a data communication protocol between pieces ofequipment [19,20]. The following object types and their attributes areused to model various types of sensor-derived operation data such astemperature, pressure, and ON/OFF status: Analog Input (AI), AnalogOutput (AO), Analog Value (AV), Binary Input (BI), Binary Output (BO),and Binary Value (BV). This combination of performance data withgeometry model could enhance readability and comprehensibility ofindividual information, especially for persons who do not have anybackground knowledge of the particular piece of equipment.

4.2.3. Design of the O&M information modelIn order to effectively support automated equipment/facility

identification and visualize equipment-specific information, theidentified information types are categorized into three groups asshown in Fig. 2. The maintenance information includes equipmentspecifications, sub-systems, and personnel who are involved inoperating and maintaining the particular piece of equipment as wellas O&M history. One fundamental information type of equipmentO&M history is the list of work orders. The operations informationrefers to mainly equipment performance data (or data stream)including sensor-based data, set points. Finally, geometric represen-tation is used to help O&M personnel better understand the status ofequipment and facilities that the equipment serves as well as theirexact location at the site.

4.3. Architecture of the AR-based O&M Fieldwork Facilitator

Fig. 3 depicts the architecture of AROMA-FF. The AR-based objectdetection manager takes live video stream and analyzes it to identifyequipment and facilities in real time. In order to deal with varioustypes of O&M information, AROMA-FF consists of the followingthree managers: the maintenance information manager, the BAS data

Fig. 2. High-level categories of equipment-specific O&M information.

342 S. Lee, Ö. Akin / Automation in Construction 20 (2011) 338–352

manager and the geometry manager. Each of these managers isintended to deal with equipment/facility maintenance information,BACnet-compliant BAS (sensor-based operation data) and equip-ment/facility geometry representations, respectively. The geometrymanager reads three-dimensional models built and imported fromCAD applications in ASE (ASCII Scene Export), OBJ and VRML (VirtualReality Modeling Language) formats.

The ARTag [21], the software library used for the prototypeimplementation, provides functions for the object detection managerand the geometry manager. The BAS data manager uses the BACnetProtocol Stack [22], an open source software library, which enablesusers to read and write values from/to BACnet objects. The dataretrieved by the BAS data manager are integrated into the informationvisualization manager (AR-based) so that the user can see equipmentperformance data with its spatial location and maintenance-relatedinformation, such as maintenance history.

Although the architecture was initially designed to interact withgeometry management databases and CMMS, it would have beenimpossible to complete the implementation of these managersbecause of limitations in the real practice. So, for validationexperiment, we had to provide geometry data and CMMS data fromthe local computer used to run the application.

Fig. 3. System architecture of the AR-based O&M fieldwork facilitator.

4.4. Verification tests

A prototype application of AROMA-FF was tested in order to verifythat it provides the functionalities and behaves as intended. Tworesearch and educational facilities and their HVAC systems were usedfor this test: a ventilation system and a mullion system in one facility,and an Air Handling Unit (AHU) in the other facility. The controlsystems for those HVAC systems were different. The AutomatedLogiccontrols the ventilation system, while two separate Johnson ControlMetasys® are used to control the mullion system and the AHU.

This verification test was conducted with the following two parts:retrieval of operation data from HVAC systems in real time andsuperimposition of the 3D model of the mullion system. The objectiveof the first part was to test if the BAS data manager is able tocommunicate with BACnet-compliant HVAC control/monitoringsystems in real time through the Internet Protocol (IP). The secondpart aimed to see if the prototype application is able to superimposefull scale computer-generated 3D models onto the equipment of themodels in video stream in real time.

4.4.1. Retrieving sensor-based data from BACnet-compliant BASThis verification test was conducted with Metasys® of Johnson

Controls that controls AHU for a research and educational facility andAutomatedLogic's building control system that controls ventilationunits for another research facility. The data collected through theprototype application was compared with the data obtained from theproprietary interfaces provided by the HVAC system manufacturers.The test results showed that the prototype application retrieved thevalues of all the properties identified through the field observationweconducted, including Object Name and Type, Present Value, and Out ofService.

4.4.2. The superimposition test of 3D geometric modelsThe second test (superimposition of the three-dimensional

geometric representation of the mullion system) was conductedwith a bay in a research facility. Fig. 4 shows the bay and itscomponents, including HVAC systems, structure elements, wallelements, furniture, documents and electronic devices. The compo-nents of the mullion system consist of a supply pipe, a supply valve,four fin-tube pipes, a return valve and a return pipe, as shown in thediagram. The mullion supply pipe and the supply valve are below thefloor panels and four fin-tube pipes, a return valve and a return pipeare exposed to the bay. Since the components of the mullion system,structure elements and wall elements are all painted the same color, itis not easy for those who are not familiar with the facility todistinguish mullion components from others.

A 3D model of a mullion section was modeled and assigned to aphysical marker (see Fig. 5) so that the prototype application couldrecognize and use it as an origin onto which the mullion model couldbe superimposed whenever the application detected the marker. Thepipe segments of the 3D mullion model which are located below thefloor panels and unseen by occupants are colored in yellow so thatusers of the prototype application can easily recognize them as non-visible components (the figure on the right). As shown in the figure,the prototype application enables the user to locate the supply valveand the mullion supply pipe in the plenum as well as four exposedrisers that heat the space. The superimposition of themullionmodel issomehow distorted at the corner. However, the two risers in thecenter show accurate superimposition in both horizontal and verticaldirections. Furthermore, in the case of upper portion of the system(the figure on the left), the superimposition of all the mullioncomponents such as the four risers, the return pipe and the returnvalve is accurate enough for users to easily distinguish the mullioncomponents from other building elements such as structural. Thefigure also shows the result of displaying equipment-specific

Fig. 4. A IW bay and components of the mullion system.

343S. Lee, Ö. Akin / Automation in Construction 20 (2011) 338–352

maintenance information on the left side as well as displaying realtime performance data stream on the upper-right corner.

Overall, as the figures show, the prototype application was able tosuccessfully superimpose equipment geometric models in full scaleand display equipment-specific maintenance information and oper-ation data onto the real equipment objects in real time withacceptable accuracy.

5. Experimental evaluation of the AR-based O&M FieldworkFacilitator

This section describes the experimental evaluation of the AR-basedO&M Fieldwork Facilitator (or AROMA-FF) using the implementedprototype application. The main objective of the evaluation is toquantitativelymeasure the performance of O&Mpersonnel when theyuse AROMA-FF during their fieldwork. For the evaluation, a researchand educational facility in the institution studied in the research wasmodeled in a Virtual Reality environment and the experiment wasconducted with FMS (Facilities Management Services) personnel andan O&M staff member primarily in charge of operating andmaintaining the facility and its building systems. The analysis of theexperiment results focuses on improvement in efficiency through theprototype application during O&M fieldwork. The primary metrics are

Fig. 5. A mullion section and its 3D mod

the amount of time spent on O&M activities and the transit instancesto complete a work order. It was expected that comparingexperimental results when the subjects used AROMA-FF againstthose obtained when the subjects used conventional strategiesnormally used in the current O&M practice would highlight improve-ments in efficiency by improving contents and accessibility of O&Minformation.

5.1. Experiment plan

5.1.1. Facilities and equipmentA research and educational facility selected for this experiment

consists of offices, conference spaces and control rooms (see Fig. 6). 12office modules (bays 1 through 12) and the local control room wereused for this experiment. HVAC systems are controlled through BAS,which are easily accessible. These systems include the ventilation(SEMCO Revolution unit) and mullion systems, which were selectedfor this evaluation. The mullion system is the primary heating systemof the facility. Johnson Controls Metasys® monitors and controls thesystem. SEMCO revolution unit® controlled through AutomatedLogicWebCtrl® serves to ventilate the facility and uses BACnet tocommunicate with other devices. Fig. 7 depicts the hot water systemof the facility and sensors deployed tomonitor and control the system.

el superimposed onto the section.

Fig. 6. The floor plan of the research facility.

344 S. Lee, Ö. Akin / Automation in Construction 20 (2011) 338–352

This system provides hot water to mullion system and the ventilationsystem as a primary energy source.

5.1.2. ScenariosSix different scenarios were developed to simulate various

representative Operations and Maintenance situations. They can begrouped into two categories: hardware repair and adjusting operationvariables to increase/decrease indoor air temperature. The scenariosof hardware repair focus on measuring tradespeople' performancewhile conducting repair fieldwork. In this set of scenarios, the subjectconfronted hardware problems with pipes and two types of valves (amanual return valve and an automatic supply valve with an actuator)while diagnosing the mullion system. Three specific hardwareproblems are developed as follows: leaky supply valve, cloggedmullion pipes and missing control actuator of the supply valve. Theother set of scenarios represents fieldwork situations in which O&Mpersonnel adjust equipment-setting values to satisfy occupants. Thethree scenarios are adjusting mullion supply valve opening, adjustingmain hot water supply valve, and adjusting the temperature of supplyair from the ventilation system. In these scenarios in particular,subjects first try to adjust values of the supply valve opening for thelocal area to increase or decrease hot water supply in order to increaseor decrease indoor air temperature of the particular problem area. Itwas expected that subjects would try to adjust the supply valveopening and see if temperature sensors showed correspondingtemperature values. It was also anticipated that subjects wouldcheck if the occupants of the bay were satisfied with the change. Incase the occupant was not satisfied with the result, the subject couldthen increase/decrease the main hot water supply valve of the facility,which feeds hot water to the facility HVAC systems. Subjects neededto adjust set points of the ventilation system to increase the roomtemperature when the mullion system, the primary heating system ofthe facility, operated at its maximum capacity.

Fig. 7. A schematic diagram

5.1.3. SubjectsThe primary prospective users include staff members in FMS

department who operate and maintain equipment and facilities.These subjects were recruited on a voluntary basis from the FMSpersonnel (one general O&M manager, one steamfitter, three A/Cmechanics and one plumber) and one O&M staff member for thefacility used in this evaluation. The subjects' years at the institutionvary from recently hired to more than 20 years' experience. However,the subjects have enough background knowledge in their specific fieldand the O&M systems used for the institution to perform their jobsadequately. Consequently, there was no significant inefficiency factordue to a lack of basic training from the perspective of either theirdomain knowledge, or knowledge about facility-oriented O&Msystems.

5.1.4. The evaluation environmentThe experiment was conducted in a two-walled immersive Virtual

Reality (VR) environment (see Figs. 8 and 9) instead of an actualfacility in the real world due to the limitations of the experimentenvironment and the benefits of the VR environment. The limitationsinclude insufficient flexibility in customizing the facility and BAS thatserves the facility for the experiment and interference between theexperiment and occupants' work in the facility. Building amockup of afacility, an alternative way to set up an experiment environment, isvery costly. A realistic 3D VR world, however, gives all the freedom ofcustomization and the experiment can be conducted without anyinterference with occupants in the actual facility. Moreover, when theuser uses AR application with Head Mount Display (HMD), he or shesees computer-generated video stream that the AR applicationmakes.As the purpose of this evaluation is to see the enhancement ofefficiency in providing information for O&M fieldwork, there is nosignificant difference between the real facility and its VR model.

As shown in Fig. 9, when the subjects conducted given scenarioswith conventional methods, they navigated the virtual world of the

of the hot water system.

Fig. 8. Hardware configuration of the experiment.

345S. Lee, Ö. Akin / Automation in Construction 20 (2011) 338–352

facility directly by seeing the VR screens. When the AR prototypeapplication is used, the subjects looked at the monitor screen wherethe AR prototype application displayed video stream of the virtualworld captured through the web-camera and processed withcomputer-generated data associated with the particular visualmarkers. Screenshots are shown in Fig. 11.

The Unreal Tournament 2004® (software application) was usedto model and simulate the three-dimensional virtual world of thefacility. The CaveUT, developed by Jacobson and his research team[23], was used to enable the game software application to renderand display three-dimensional views on the immersivemulti-screenVR environment. The prototype application and the VR environmentare two independent applications. They do not communicate witheach other. The VR environment only displays the VR world of thefacility and the prototype application analyzes video stream thatcaptures the scene displayed on the VR world and superimposescomputer-generated data in response to the user's input in realtime.

Fig. 9. Hardware configuration of the VR environment and

5.1.5. MetricsThe metrics are time spent on activities to obtain information and

the total number of transit instances to complete a work order. Thetime spent on each activity was measured and the time data wascompared to estimate how much the prototype application can savetime on performing fieldwork activities over various conventionalstrategies used in the current O&M fieldwork practice.

The total number of transit instances to complete a work order isanother variable. Although transit is an essential O&M activity, it isone of the most common non-value-adding activities. Therefore,throughout this experiment, we expected to see if the prototypeapplication could reduce the number of transit during their fieldwork,resulting in improving fieldwork efficiency.

5.1.6. The experimental procedureThe subjects conducted one hardware repair scenario and one

operational adjustment scenario with the prototype application andanother set of scenarios using conventional strategies with which to

the VR model of the facility projected on the screens.

346 S. Lee, Ö. Akin / Automation in Construction 20 (2011) 338–352

get information. Each scenario started by giving the subject a workorder. While conducting a scenario, the subject was asked to verballyexplain what activities he wanted to perform at the moment so thatthe investigator could understand what the subject was doing andproperly respond to the activities. Similarly, the subject and theinvestigator verbally communicatedwith each otherwhen the subjectwanted to talk to other personnel to get information. In that case, theinvestigator played the role of the person the subject wanted to talkto. The scenario ended when the subject returned to the entrancepoint of the virtual world, declared that he had completed thefieldwork and then proceeded to complete the work order documen-tation. The sequence of each subject's O&M activities was recorded inthe order of occurrence, as shown in Table 1. The total number of 28records was collected.

The information types and instances used for the experiment werecollected from the existing O&M systems, written documents,drawings and results from interviews with O&M staff members ofthe facility. For the scenarios in which the subjects did not use theprototype application, each type of information was provided in thesame format as it was obtained except for performance-related data.These data were properly determined for various O&M situationswhen the experiment was designed and provided when the subjectasked during the experiment.

As mentioned earlier, performance-related data were pre-deter-mined in the experiment design phase and the data were verballyprovided immediately at the subject's request. Consequently, timespent on operating software applications to get performance-relateddata was not measured. In order to measure actual time spent, theinvestigator measured the time spent on operating the actualapplications to access the BAS of the facility and retrieve the data.The measurement was repeated 20 times for each data type that was

Table 1Example of activity sequence record with time and location data.

Scenario: change main hot water supply temperature using the prototype application

Action

Read work orderChecks if pumps are running properly through the FMS control systemChecks incoming steam temperatureMoves to the facility entranceMoves to the bay #10 (interface superimposes the bay number)Moves to the bay #7 (interface superimposes the bay number)Moves to the bay #8 (interface superimposes the bay number)Confirms to the occupant that the office is too coldReads the mullion section information superimposed on the interfaceGets the surface temperature of the mullion section through the prototype applicationBleeds air from the mullion pipesMoves to the facility control roomGets the position information of the control valve in the bay #8 through the interfaceand the facility control system

Moves to the facility mechanical roomChecks if the mixing valve is working properlyChecks water flow, the temperature of hot water supplyChecks if the pump is onGet main mullion supply water temperature and return water temperatureDecides to raise the supply water temperatureMoves to the facility control roomGets the local staff's phone numberExplains to the local staff that the subject will ask to change hot water temperature set-pin order to check if it remedies the problem

Exchanges phone numbers each otherLocal staff changes the set-point of hot supply water temperature to 110 F and the interfdisplays the temperature value

Moves to the bay #8Gets surface temperature of the mullion section through the interfaceGets room temperature of the bay #8 through the interfaceCalls the local staff to maintain the current set-point valueMoves to the facility entranceDocuments the work order

captured from the experiment, and average values were calculatedfrom the results. These average time data are used for the activities ofobtaining sensor-derived operation data when time spent onobtaining the data using the prototype application is compared tothe time using conventional strategies.

As the entire experiment was conducted using the VR environ-ment, the transit time measured during the experiment is differentfrom the actual transit time in the real facility. In order toquantitatively estimate the potential time saved from O&M activitiesassociated with transit, the investigator also measured the actualtransit time from one place to another in the real facility. Theinvestigator repeated this measurement for 20 times and averagevalues were calculated. These average time data were used forcomparative analyses of time spent on obtaining information usingthe prototype application against conventional strategies.

The time data in Table 1 are calculated using the estimated averagetime data. For example, the transit time from the facility entrance tothe bay #10 (the fifth activity) is 19 s. This time data is an averagetransit time estimated in the real facility. Similarly, 44 s wereestimated to get the surface temperature of the mullion sectionthrough the prototype application (the 10th activity).

5.1.7. Experiment scope and limitationsThe scope of the experiment includes target equipment, target

tasks, and primary actors. Target equipment includes pipes, ducts andtheir valves, which are installed in three-dimensions to convey waterand air for heating/cooling and ventilation. Target tasks are dailymaintenance and operation work commonly requested by occupants.Primary actors are tradespeople who actually take care of the O&Mrequests. Therefore, throughout the experiment, we expect to see a

Time(seconds)

Location O&M activity category

7 FMS Read documentN/A DiagnosisN/AN/A Transit19854 Bay Diagnosis

1144

N/A Repair16 Transit52 Control room Diagnosis

28 TransitN/A Mechanical room DiagnosisN/AN/AN/A12 Decision28 transit3 Control room Exchange O&M

informationoint 16

4ace N/A Repair

16 Transit29 Bay Inspection (bay)1812 Exchange information16 Transit38 FMS Documentation

347S. Lee, Ö. Akin / Automation in Construction 20 (2011) 338–352

potential for saving time while tradespeople conduct fieldworkactivities to take care of common O&M requests.

The limitations of the experiment are from the VR environmentwhere the experiment was conducted and also from the prototypeapplication. As the experiment was conducted in the VR environment,the results of the experiment cannot directly reflect the real world. Inthis research, we estimate a potential enhancement through theprototype application in saving time by using time data measured inthe real facility based on the experiment results during the post-experiment phase. In addition, the findings are limited by the facilityand the scenarios used for the experiment. Because the experimentwas conducted with the specific scenarios in the particular facility, theresults from the experiment, especially the activity of locating targetbays, can be significantly different from those in other facilities.

5.2. Comparative analysis of time spent on O&M activities toobtain information

In order to realize time saving when the prototype application wasused, time that the subjects spent on fieldwork activities using theprototype application is compared with the time that the subjectsspent when they used conventional strategies to obtain informationwhile conducting scenarios. Conventional strategies refer to methodsof obtaining information used by the subjects when they conductedO&M scenarios without the prototype application. They were not pre-determined, but identified from the experiment results in the post-experiment phase.

As described in Section 5.1.6, it is apparent that the transit time inthe VR environment could be noticeably different from actual transittime in the real facility. In fact, the transit time measured in the VRenvironment is less meaningful than the estimated transit timemeasured in the real facility because each subject's control skills innavigating the VR environment influenced the transit time in the VRenvironment, which significantly reduces the reliability of the time. Inorder to eliminate the difference, the transit time in the VRenvironment is substituted with estimated transit time measured inthe real facility by the investigator, as shown in Fig. 10. It isnoteworthy that in this study we focus on time savings whenfieldworkers try to obtain information using the prototype applica-tion, a computational approach to provide Just-In-Time information atthe site, against conventional methods as they carried out work orderscenarios. The impact on the overall O&M fieldwork when theprototype application is used needs further investigation.

Fig. 10. Procedure to calculate estim

The comparison results are categorized into three groups ofinformation types: three-dimensional geometric models with textinformation (used to locate equipment and space), operation-relateddata stream from sensors, and equipment-specific maintenanceinformation, including Person, Equipment, O&M history, and Spareparts for the specific equipment. Fig. 11 shows screenshots of theprototype application displaying these types of O&M information.

5.2.1. Comparison of time spent on locating the target areaThe subjects used three conventional strategies to locate target

areas. The first strategy was visiting bays and questioning occupantsuntil the target bay was found. The second was getting the bay plan ofthe facility at its control room and finding the location of the targetbay. The last strategy was a hybrid of the two strategies. In otherwords, the subjects failed to locate the target bay with the firststrategy, so they had to use the second strategy. When the subjectsused the prototype application, their strategy was to just visit with thebay ID superimposed on the video stream, as shown in Fig. 11. Thestrategy of using a hardcopy of the bay plan and the prototypeapplication both use the geometric representation of the facility andequipment. The differences are how to obtain the information andwhether or not additional interpretation of the information isnecessary. The prototype application automatically detects the facilityand visualizes the geometric representation in full scale, making itunnecessary for the user to perform extra activities such as transit andinterpretation of the information.

Table 2 and Fig. 12 show the estimated time spent on locatingtarget bays and their averages for comparison between the twomethods, respectively. As shown in Fig. 6, the distance of each baygroup from the facility distance is different. As described earlier,average transit time data measured in the real facility are used toreflect real facility conditions. Each subject conducted two scenariosof locating target bays using conventional strategies, listed in thetable, and another two scenarios using the prototype application. Thetime data listed in the tableweremeasuredwhen the user reached thetarget bay from the facility entrance at the beginning of each scenario.The number in front of the parentheses is the time spent on findinginformation about the location, such as getting and reading the facilitybay plan and questioning occupants. The numbers in parentheses areestimated time spent on transit from one place to another.

The results show that the time spent on locating target bays usingthe prototype application is significantly less than the time using theconventional strategies. When using the prototype application, the

ated time getting O&M data.

Fig. 11. Screenshots of AROMA-FF superimposing various O&M information.

348 S. Lee, Ö. Akin / Automation in Construction 20 (2011) 338–352

average time was 34 s, which is 48% greater than the time using thevisiting-and-questioning strategy, and 53% and 75% less than theresults using the other two strategies, respectively. Overall, using theprototype application took 51% less than using the conventionalstrategies. In several cases, the visiting-and-questioning strategy tookless time than using the prototype application. Thus, it appears to bethe best strategy in this experiment. However, the subjects used thehybrid strategy (visiting-and-questioning and using the bay plan)because they first failed at the visiting-and-questioning strategy.These results mean that using the visiting-and-questioning strategycan easily become an unreliable method to locate facilities. However,the prototype application can prevent this kind of failures.

5.2.2. Comparison of time spent on obtaining sensor-based operationdata from BAS

This section compares the results of time spent on obtainingvarious sensor-derived operation data from the BAS. The time datawere collected from the activity records (see Fig. 10 and Table 1). Thisdata includes discharge air temperature, mullion supply valveopening percentage of size, mullion surface temperature, hot watersupply temperature, and indoor air temperature. There were twodifferent building automation systems: one was to monitor andcontrol hot water supply and return, and the other was to controlindoor air ventilation. These two systems have their own proprietaryinterfaces.

Table 2Time spent on locating the target bay with estimated transit time (in seconds).

Conventional strategies Prototypeapplication

Visiting baysand questioningoccupants (1)

Using hard copyof bay plan (2)

Hybrid of (1)and (2)

Bay 3, 4 8 (+20) 61 (+74),48 (+90)

27 (+9)

Bay 5, 6 15 (+33) 0 (+12)Bay 7, 8 0 (+16) 0 (+16),

0 (+32)Bay 9, 10 27 (+21),

164 (+18),4 (+44)

0 (+25)

Bay 11 24 (+38),35 (+38)

0 (+41)

Bay 12 0 (+26) 17 (+24) 60 (+26),0 (+26)

Average 23 72 137 3470

The conventional strategy which the subjects used is to visit thelocal control room and ask the staff of the facility to retrieve data fromthe BAS using the proprietary software applications, which meansthat the total time obtaining these types of data consists of the transittime and the time operating the software applications to retrieve thedata. Similar to the previous comparison, average estimated transittime and estimated time obtaining operation data are used for thiscomparison. When the subjects use the prototype application, theprototype application retrieves data streams from BAS and displaysthem in the graphical format as well as the latest value. Transit is notnecessary when the subjects use the prototype application.

Fig. 13 shows the results, where ‘C’ in the figure refers toconventional strategies. The average time purely operating theproprietary interfaces to get operation-related data was 33 s, which isnot significantly different from the time using the prototype application(34 s). In fact, except for the case of obtainingdischarge air temperature,the time obtaining the rest data types using the proprietary applicationswas less than the time using the prototype application. However, whenthe transit time, which was necessary for the conventional strategy, isadded to the time spent using the conventional strategies, the averagetime increases to 38 s, which is 4 s (or 11.7%) more than the averagetime with the prototype application.

Although it is not statistically clear whether there is a significantdifference between using conventional strategies and using theprototype application in obtaining sensor-based operation data fromBAS, the results reveal that the subjects tended to visit the placewhere they could get the data and had the appropriate person retrievethe data for them. These types of conventional strategies requiretransit from the subject's current location (i.e., the problem site) totheir destination (i.e., the control room of the facility), and are themost common non-value-adding activity revealed through theexperiment, which is consistent with the results of the Lee and Akin'sshadowing experiment [7]. Therefore, minimizing or eliminatingtransit is the key challenge in improvement in fieldwork efficiencywhen obtaining sensor-based operation data.

5.2.3. Comparison of time spent on obtaining equipment-specificmaintenance information

The types of equipment-specific maintenance information provid-ed for this experiment were ‘detailed equipment’ information,information on ‘O&M history’, ‘person’, and ‘spare parts’. As described,the time data used for this comparison was collected from the set ofuser's activity records (see Fig. 10 and Table 1) and estimated averagetransit time measured in the real facility is used for correspondingtransit activities. The conventional strategies used to obtain spare

Fig. 12. Bar chart of average time locating target bays using conventional strategies and prototype application (unit: seconds).

349S. Lee, Ö. Akin / Automation in Construction 20 (2011) 338–352

parts information include reading documents about the particularequipment and verbal communication with the O&M staff member ofthe facility (the investigator played the role).

Table 3 shows the time data spent on obtaining spare partsinformation and O&M history information. The number in front of theparentheses is the time spent on obtaining the particular informationand the number in parentheses is the estimated time spent on transitfrom one place to another. The average time spent on obtaining spareparts information using the conventional strategies was 69 s, and 38 susing the prototype application, which is time saving of 45%. Theconventional strategies expected to observe when the subjectsobtained O&M history information include reading previous workorder sheets and questioning co-workers. However, there was onlyone observation of talking with their co-workers, whose role wasplayed by the investigator, which took 21 s. There were threeobservations of getting O&M history from the simulated CMMSusing the prototype application: two observations of getting the mostrecent work order (81 and 111 s) and one observation of getting allprevious work orders associated with the target equipment (138 s).For the other types of information, there was no observation of eitherusing the prototype application or using conventional strategies.Therefore, the time data from the experiment is not big enough forstatistical comparison. However, the results reveal that the subjects

Fig. 13. Bar chart of average time obtaining sensor-based operation

did not use maintenance information frequently even though theinformation is accumulated into their CMMS throughout the lifecycleof buildings and is easily accessible.

Databases were infrequently used to get information on equip-ment and parts. This has two main causes: the subjects' backgroundknowledge about the equipment, and the institution's standardizationof the equipment and parts. The subjects who did not usemaintenance information under any format said that they recognizewhat it is when they see the actual equipment and parts, and that theactual object is the best source for information accuracy. In addition,when information from the actual equipment was unavailable, at leastthey could narrow the number of reasonable possibilities significantlydue to the standardization. For example, the standard size of fin-tubepipes in the institution studied in this research is three-quarter inches.If subjects get a work order for hardware repair such as a water leak,they could bring to the site the standard size valves and pipe segmentsin advance and repair the faulty equipment immediately at the sitewithout going back to the FMS shop to get the required materials.Furthermore, the subjects already had information on specific peoplesuch as primary contacts and cell phone numbers either in memory orin their portable devices such as a cell phone. Therefore, the subjectsdid not need to access the central databases unless the recall of thisinformation was different, in which case the subjects would call

data using conventional strategies and prototype application.

Table 3Comparison of time spent on obtaining spare parts information and O&M historyinformation (unit: second).

Conventional strategy Prototype application

Time onobtaining data

Average Time onobtaining data:

Average

Spare partsinformation

60 (+3),64 (+11)

69 29, 46 38

O&M historyinformation

21 21 81, 111, 138 110

Table 5The number of transit instances to complete a work order of operations variableadjustment (P: Prototype application and C: Conventional Strategies).

Scenarios Adjust mullionsupply valveopening of a bay

Adjust supplyhot watertemperature

Adjustdischarge airtemperature

Strategy P C P C P C

Subject1 6 7Subject2 8 16 (6)Subject3 6 6Subject4 6 6Subject5 4 7Subject6 6 9Subject7 20Average 6 6 9.75 10.33 6 6.5

350 S. Lee, Ö. Akin / Automation in Construction 20 (2011) 338–352

others until they got the correct information. Finally, no tradespeople(AC mechanics, steamfitters and plumbers) asked for O&M historyinformation during the experimental scenarios. They just focused onremedying the given problem. The role of the subject who collectedthe background information about maintenance history was an O&Mgeneral manager whosemain job at the site was to collect informationabout the facilities and equipment conditions and make properdecisions. It would be natural for the subject to collect informationfrom foremen and tradespeople in the diagnosis phase. However,since the tradesperson's role is to execute the instructions from hismanager and/or foreman, it would not be a top priority task to obtaininformation about the equipment background.

5.2.4. Comparison of the number of transit instancesIn order to see whether or not the subjects were able to reduce the

number of transit instances during experimental fieldwork, which isthe number of transit instances that occurred to complete a workorder scenario when using conventional strategies and the prototypeapplication was compared. Tables 4 and 5 show the number of transitinstances performed by each subject to complete work orders ofHardware Repair and Operations Variable Adjustment with the twodifferent strategies of obtaining O&M information (i.e., usingconventional strategies (C in the table) and using the prototypeapplication (P in the table)). The numbers in parentheses refer to thenumber of transit instances needed for the sole purpose of obtaininginformation, which could be eliminated when proper computationalsupport such as the prototype application is used. For example, toobtain O&M information, Subject2 performed two transit activitieswhen he conducted the scenario with the prototype application andthree transit activities when he used conventional strategies, whichwere redundant transit activities.

As shown in Fig. 14, the results of the experiment do not show aclear indication of a decrease in the number of transit instances withthe prototype application over conventional strategies, although inmost scenarios, the number of transit instances to complete a workorder using the prototype applicationwas slightly less than or equal to

Table 4The number of transit instances to complete a work order of hardware repair(P: Prototype application and C: Conventional strategies).

Scenarios Replace equipment components Unclog mullionpipes

Replace missingactuator (S1)

Replace leakysupply valve (S2)

Strategy P C P C P C

Subject1 10 6 (2)Subject2 16 (3) 10 (2)Subject3 9 (2) 9Subject4 10 (3) 8 (2)Subject5 8 8 (2)Subject6 12 9 (3)Subject7 9 8Average 10.33 11.67 8.67 8 8 8

the number of transit instances using conventional strategies. Onepossible explanation for this is that the results are more dependent ona personal preference established through their fieldwork experience.As shown in Table 4, in 4 out of 7 cases of hardware repair scenarios,the subjects performed transit activities just to obtain O&Minformation, even though they could get the information remotelyat the site through the prototype application. Furthermore, thesubjects who conducted the scenario of replacing the leaky supplyvalve using the prototype application performed, on average, morenumber of transits than when using the conventional strategies. Inthis particular scenario, two out of three subjects performed transitfor information purposes during the scenario, which could have beenavoided if they used the prototype application instead. We considerthis a personal preference, which does not necessarily target moreefficient operations. These results show that at this point providinginformation when requested is not a dominating factor in enhancingan individual O&M fieldwork flow, especially for daily maintenance,for which O&M personnel can control their schedules and time at thesite as long as they can complete work orders by a given due date.

5.3. Implications of AROMA-FF on O&M fieldwork efficiency

The experiment results revealed that the prototype applicationshowed the potential for saving time through information supportduring O&M fieldwork. On average, the prototype application enabledthe subjects to locate the target bay more than 50% faster than theconventional methods they used. According to Lee and Akin [7], thetradespeople spent 6% of their time locating equipment and facilities.For this one activity, if 50% time saving observed in the validationexperiment is applied to the result, 3% of the tradespeople's time atthe site could have been saved. Moreover, as the information relatedactivities and transit are highly positively correlated with O&Mactivities [7], further improvement in fieldwork efficiency can beachieved by saving time spent on irrelevant O&M fieldwork activities.For example, during the validation experiment, we observed a case inwhich the subject diagnosed the wrong bay and concluded that therewas no problem with the building systems, resulting in a more than50% time loss before he was able to diagnose, repair and inspect thecorrect bay. The reason for this is that the subject relied on hisincorrect recall information on bay numbers. In the validationexperiment, there were five cases of visiting incorrect bays, whichwere immediately corrected by the prototype application withsuperimposed bay numbers.

Furthermore, transit time can be saved with proper informationsupport. The impact of the extra transit time from the results of thevalidation experiment appears insignificant because, for example, theaverage time spent on extra transit to obtain sensor-based operationdata was just six seconds, considering the total time spent on

Fig. 14. Bar chart of transit using prototype application and conventional strategies.

351S. Lee, Ö. Akin / Automation in Construction 20 (2011) 338–352

completing a work order. However, this is mainly because theexperimental environment was relatively small (its dimensions are1701 ft×687 ft. On average, it only took 28 s to move from theentrance to the facility control room, which is located in the oppositeside of the entrance). If the experimental environment were extendedbuilding-wide, the transit time would increase significantly. Theoverall fieldwork efficiency can be improved with additionalimprovement from the correlated core maintenance activities, whenthe extra transit is eliminated by providing information at the sitethrough a computational fieldwork support application such asAROMA-FF. From the experiments, it is clear that AROMA-FF canhelp in locating equipment and facilities and has a potential forfurther time savings in obtaining sensor-based operation data,resulting in the improvement in fieldwork efficiency not only bydirectly saving time spent on specific O&M activities, but also bypreventing potentially catastrophic inefficiency.

6. Conclusions

It has been a well-known problem in the O&M practice to collectvarious types of data to locate the target equipment and facilities andto properly diagnose them at the site. The objective of this researchhas been to provide Just-In-Time O&M information through the use ofcomputational support resulting in improving O&M fieldworkefficiency. An Augmented Reality-based O&M Fieldwork Facilitator(AROMA-FF) was developed and evaluated to see the improvement infieldwork efficiency when the integration of various O&M informationtypes and visualization of those information using Augmented Realitytechnology. The results of the validation experiments show that theapplication allows the user to locate target areas significantly fasterthan conventional methods such as using the facility bay plan.Although there is no significant difference between AROMA-FF andconventional strategies when the fieldworkers obtain sensor-basedoperation data due to automated BAS environments, the time savingcould be attained when the control room where to get the data is farfrom the site because the difference is due to mainly transit. Thefindings, therefore, reveal a potential for improvement in O&Mfieldwork efficiency by reducing or eliminating redundant andexcessive activities due to information related difficulties as well asnon-value-adding activities, such as transit. In addition to the findingof time savings, further possible benefits from the application includethe intuitive use of the digital model of a building, which frees thefieldworker's both hands increasing their safety in extreme jobconditions, intelligent visualization of the current status of equipment

and facilities by reasoning from various data retrieved from differentbuilding automation systems and O&M history databases.

7. Limitations and future work

The limitations of the current prototype application that need to beovercome are listed as follows. As the computer vision-based ARtechnology that recognizes pre-defined physical markers is used toidentify objects, the application cannot recognize objects when themarkers cannot be clearly seen. Besides, it is not realistic to deployphysical markers to equipment and building elements. The userinterface supports only keyboard and mouse inputs, which couldreduce the usability of the application significantly. Better user-inputmethods such as voice command recognition need to be investigatedto overcome the limitation. In addition, the output interface of thecurrent application is not designed to support small mobile devicessuch as smart phones or headmount displays. Further development tosupport such devices is necessary to improve the usability of theapplication. Furthermore, the experiments of this study wereconducted in a virtual reality environment. The findings from theexperiments show the potential of the application. As the goal of theapplication is to support O&M fieldwork activities, further experi-ments in real facilities is critical for its validation.

This study can be extended in various ways, including improve-ment in the technologies to overcome the limitations stated earlier aswell as broadening its scope. In order to automatically identifyequipment and facilities and provide information associated with theparticular equipment and facilities at the site, different technologiessuch as barcodes, RFID, GPS and Wireless Sensor Network need to beinvestigated to find the most suitable method for the particular siteconditions. In order to effectively support situation-based O&Mfieldwork activities in real time, methods for management of differenttypes of information as well as manipulation and display ofequipment-specific information and performance data need to beinvestigated as the size of building information models, maintenancehistory information and performance data collected from numeroussensors throughout the lifecycle of a building are frequently too big tohandle in real time. From the perspective of application domains,emergency response is another promising application for AROMA-FF.Certain types of emergencies such as flood and power outage arehighly related with building systems. Response to these types ofemergencies is one of the most extreme situated O&M cases andresponse time is critical to protect occupants as well as importantequipment and facilities. As such, timely information support plays an

352 S. Lee, Ö. Akin / Automation in Construction 20 (2011) 338–352

essential role in halting the problem and diagnosing equipment andidentifying causes for the emergency.

Acknowledgement

This research is underwritten by the National Institute ofStandards and Technology, grant numbers 60NANB5D1161 and60NANB6D6157.

References

[1] Federal Energy Management Program, Operations & Maintenance, http://www1.eere.energy.gov/femp/operations_maintenance/index.html2007.

[2] L. Mann Jr., Maintenance Management, Revised Ed., Lexington Books, Lanham,1984.

[3] B. Chanter, P. Swallow, Building Maintenance Management, Hartnolls Ltd.,London, 1996.

[4] D.G. Cotts, M. Lee, The Facility Management Handbook, AMACOM, New York,1992.

[5] E. Teicholz, Bridging the AEC Technology Gap, IFMA Facility Management Journal,http://www.bricsnet.com/content/Teicholz.pdf 2004.

[6] G.P. Sullivan, R. Pugh, A.P. Melendez, W.D. Hunt, Operations & Maintenance BestPractices: A Guide to Achieving Operational Efficiency Release 2.0, http://www1.eere.energy.gov/femp/pdfs/OandM.pdf 2004.

[7] S. Lee, Ö. Akin, Shadowing tradespeople: inefficiency in maintenance fieldwork,Automation in Construction 18 (5) (2009) 536–546.

[8] G.L. Thomas, Operations and maintenance, in: TeicholzE. (Ed.), Facility Designand Management Handbook, McGraw-Hill, New York, 2001, pp. 22.1–22.42.

[9] E.P. Rondeau, R.K. Brown, P.D. Lapides, Facility Management, John Wiley & Sons,Inc., New York, 1995.

[10] E. Teicholz, Facility Design and Management Handbook, McGraw-Hill, New York,2001, pp. 25.15–25.17.

[11] L.Y. Liu, A.L. Stumpf, S.S. Kim, F.M. Zbinden, Capturing As-built Project Informationfor Facility Management, Proc. of the First Congress held in conjunction with A/E/C Systems '94, ASCE, New York, 1994, pp. 614–621.

[12] M.J. Clayton, R.E. Johnson, Y. Song, J. Al-Qawasmi, A Study of Information Contentof As-Built Drawings for USAA, http://archone.tamu.edu/crs//documents/pub-lications/asbuilt.pdf 1998.

[13] IFC Model Based Operation and Maintenance of Buildings, http://cig.bre.co.uk/iai_uk/iai_projects/ifc-mbomb/ 2004.

[14] A.J. Arnold, J.C. Kunz, Integrating product models with engineering analysisapplications: two case studies, Artificial Intelligence for Engineering Design,Analysis and Manufacturing 14 (2000) 137–147.

[15] A. Hammad, J.H. Garrett, H.A. Karimi, Location-based computing for infrastructurefield tasks, in: H.A. Karimi, A. Hammad (Eds.), Telegeoinformatics: Location-BasedComputing and Services, CRC Press, Boca Raton, 2004.

[16] S. Goose, S. Güven, X. Zhang, S. Sudarsky, N. Navab, PARIS: Fusing Vision-basedLocation Tracking with Standards-based 3D Visualization and Speech Interactionon a PDA, http://graphics.cs.columbia.edu/people/sinem/papers/PARIS-DMS2004/S.Goose-DMS2004.pdf 2004.

[17] Knowledge Based Systems, Inc., Draft Federal Information Processing StandardsPublication, http://www.idef.com/pdf/idef0.pdf 1993.

[18] Ö. Akin, M.T. Turkaslan-Bulbul, I. Gursel, J.H. Garrett Jr., B. Akinci, H. Wang,Embedded commissioning for building design, Proc. of International Conferencefor Enhanced Building Operations, Paris, 2004.

[19] American Society of Heating, Refrigerating and Air-Conditioning Engineers.ASHRAE STANDARD 135–2004: BACnet - A Data Communication Protocol forBuilding Automation and Control Networks, American Society of Heating,Refrigerating and Air-Conditioning Engineers, Inc, Atlanta, 2004.

[20] S.T. Bushby, BACnet™: a standard communication infrastructure for intelligentbuildings, Automation in Construction 6 (5–6) (1997) 529–540.

[21] M. Fiala, ARTag Revision 1. A Fiducial Marker System Using Digital Techniques,National Research Council of Canada, NRC 47419, http://www.iit-iti.nrc-cnrc.gc.ca/iit-publications-iti/docs/NRC-47419.pdf 2004.

[22] BACnet, BACnet: Developer Help, http://www.bacnet.org/Developer/index.html2008.

[23] J. Jacobson, CaveUT 2004 V1.1, http://publicvr.org/ut/CaveUT.html 2008.