Open Source EyeTap: EmpoweringEvery Maker with PhenomenalAugmented Reality and WearableComputing

Sarang NerkarUniversity of TorontoToronto, ON M5S 3G4, Canadasarang.nerkar@mail.utoronto.ca

Max Hao LuUniversity of TorontoToronto, ON M5S 3G4, Canadahao.lv@mail.utoronto.ca

Sen YangUniversity of TorontoToronto, ON M5S 3G4, Canadasen@eyetap.org

Alex PapanicolaouUniversity of TorontoToronto, ON M5S 3G4, Canadaalexp@eyetap.org

Cindy Jinhee ParkUniversity of TorontoToronto, ON M5S 3G4, Canadacindy@eyetap.org

Steve MannStanford UniversityStanford, CA 94305, USAmann@eyetap.org

Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for components of this work owned by others than ACMmust be honored. Abstracting with credit is permitted. To copy otherwise, or republish,to post on servers or to redistribute to lists, requires prior specific permission and/or afee. Request permissions from permissions@acm.org.

ISWC '17, September 11–15, 2017, Maui, HI, USA © 2017 Association for Computing Machinery. ACM ISBN 978-1-4503-5188-1/17/09...$15.00 https://doi.org/10.1145/3123021.3123075

AbstractAugmented reality and wearable computing developmenthas skyrocketed in the consumer product domain in thepast few decades, while the open source domain remainedrather neglected. We believe that this is due to the existingdevelopment platforms being expensive, closed-source anddue to the lack of a strong open source augmented realitycommunity.

We present a hardware and software open source wearableaugmented reality platform which enables users to makeaugmented reality glasses that cost less than $250 and re-alize new applications of augmented reality which can thenbe added to the platform for others to use and contribute to.

Author KeywordsMediated Reality; Augmented Reality; Wearable Comput-ing; EyeTap; Open Source, Open Innovation; Maker; Devel-oper; Tinkerer; Open Source Community

IntroductionAugmented reality technology enhances the natural percep-tion of humans by enabling them to sense and interact withthings that they normally wouldn‘t be able to, thus augment-ing to the natural capabilities of humans[17]. In his child-hood in the 1970s, Mann invented phenomenological aug-mented reality[10], wearable computing, and quantimetric

203

ISWC '17, SEPTEMBER 11–15, 2017, MAUI, HAWAII, USA

self-sensing (later called “quantified self”) which he broughtto MIT in the early 1990s to found the MIT wearable com-puting project (http://wearcam.org/nn). The miniaturizationof the overall system enabled users to wear this systemoutside the laboratory environment and use it on a daily ba-sis[18]. Mann later used this work to co-found InteraXon[4]and Metavision (Meta Company)[6].

Figure 1: Top: One of world‘s firstwearable augmented realitycomputer systems developed byMann. Middle: The EyeTapsystem. Bottom: Mann‘s recentEyeTap, completed in 1998 (19years ago), which he wears on adaily basis.

Figure 2: EyeTap principle as in [9]

Recent implementations of augmented reality are focusedmostly on overlaying images onto reality. However, the ac-tual essence of augmented reality isn’t limited to just that, itcan also be performed in forms of various seeing or hear-ing aids [12]. One example is using real time high dynamicrange video to enhance the overall dynamic range of thehuman eye [16][1][14]. Thus enabling humans to see ob-jects that are a lot brighter than what their eyes can handle.Mann developed one of the first augmented reality seeingaids, called EyeTap [11]. Since then various augmentedreality headsets have come into place such as GoogleGlass, Microsoft Hololens and Meta glasses. There hasbeen exponential development in the consumer applicationsof augmented reality glasses since Mann’s EyeTap [20] .However, there does not seem to be significant growth inthe open source augmented reality domain. There are alot of open source augmented reality software frameworksthat enable hackers, tinkerers and developers to developaugmented reality applications. But the way these frame-works are designed only enables development of con-sumer applications of augmented reality such as gamesand apps., while prohibiting enthusiasts from having thefreedom to modify or design hardware systems tailored totheir unique applications. Furthermore, these frameworksrequire the developers to buy the consumer augmented re-ality glass developer kits that cost over $1000, a cost thatmost early-stage developers cannot afford. To embody thetrue essence of augmented reality our society needs an

open source framework that is not only open source in thesoftware end but also at the hardware end, while also beingaffordable [13][7]. We are not aware of any existing solu-tions that enable the developer community to develop andcontribute to a platform. In this paper, we present a hard-ware and software open source wearable augmented realityplatform which enables the users to make augmented re-ality glasses that cost less than $250 and realize new ap-plications of augmented reality which can then be added tothe platform for others to use and contribute to.

MotivationWearable computing and augmented reality have grown ex-ponentially in the past few decades, in the consumer mar-ket domain. Although, there are various augmented realitydevelopment glasses available in the market today, theseproducts enable developers to develop games and apps forthe consumer market only. We believe that the real potentialof augmented reality hasn’t been realized. Current devel-opment has happened for people living in cities that can af-ford to spend $1000 or more on augmented reality glassesto replace their desktop computers, and this developmentis mostly closed-source. The applications of augmentedreality in rural areas, underprivileged areas and minoritycommunities have not been realized yet. We believe thatthis is because the available technology is expensive anddoesn‘t focus on the needs of the billions of people in theworld. The technology, right now, focuses on the few thou-sands who can buy these products. One way of realizingthese applications is by enabling the billions of people tofind the solutions to their problems with augmented realityand wearable computing on their own [5][2][19][3]. This canbe done by making the technology open source and provid-ing a platform where people can use and contribute to thetechnology, similar to what Linux did with operating systems[15]. This thought of having an open source platform for

204

ISWC '17, SEPTEMBER 11–15, 2017, MAUI, HAWAII, USA

augmented reality and wearable computing has been themotivation for this project.

Modularity: A New Approach in Wearable Eyeglasses

Figure 3: Open Source EyeTapdesign as part of Mobile WorldCongress special fashion show inBarcelona, 2017.

Figure 4: Base frame design withseparated frame and nose piece

We approached various maker communities to realize therequirements for augmented reality development. Whileconducting this survey we recognized certain attributes thatmake a device suitable for augmented reality development.We realized that in order for the maker community to real-ize applications in augmented reality that solve problemsin their localities, the device needs to be modular and opensource, not only in software but also in hardware. Opensource and modularity together eliminate the problem ofbeing limited by the given technology, because the makerscan just add the missing technology pieces for the develop-ment of their application. This enables open innovation, anapproach that has proven to be successful in enabling newspecialized application for various professions[3][2][19]. Wehave already seen makers develop different modules withour system for various applications such as eye tracking,clear vision at a distance and thermal vision(See figure 5).We envision that our system will enable a lot more makersto solve problems and enhance the capabilities of people intheir localities with augmented reality.

We tested different wearable eyeglasses and head up dis-plays to see how they compare with our system based onthe attributes that we realized are suitable for augmentedreality development(See figure 6). We found out that costis a major reason why modularity is not a favorable choicefor commercial products. The customization makes it lesseffective for mass production. Our system does not facethis problem because advanced tabletop manufacturingtechniques such as 3D printing allow for production of cus-tomized personal apparel such as eyeglasses.

Figure 5: Modularity enables multiple functionalities in the OpenSource EyeTap

Design ApproachThe existing EyeTaps were made from various materials in-cluding off-the-shelf safety helmet, CNC milled stainlesssteel, and injection moulded plastic as seen in figure 1.We performed an evaluation of the existing manufactur-ing methods and witnessed the benefits and drawbacks ofusing these methods. We then compared them with ourproposed manufacturing method: 3D printing. Although 3Dprinting is still at the early stages of development and adop-tion, it shows a promising future in the ergonomic related,highly customized applications like smart wearables. Withan emphasis on design for everyday use, this Open SourceEyeTap design was also used in a fashion show at MobileWorld Congress 2017, as shown in figure 3.

205

DESIGN EXHIBITION

������� ��� ��� ����������� ����� ��������� ��� ������� ������� �������� �����

� �������������

�����������

���

������������

��� � ����������� ������������� �������� !������������

�������"�#�

$�%�!���

�������#��$����� �

&��� ��������

&�����

��� ��������&�����

Figure 6: Comparison of different wearable eyeglasses and head up displays based on attributes suitable for augmented reality development

Figure 7: Evolution of opensource EyeTap. Top: Initial designusing Raspberry Pi 3, whichrequires a head strap to hold inplace. Middle: Intermediate designwith ear hook and a rail for easyassembly and micro-adjustment ofcomponents. Bottom: Currentdesign using Raspberry Pi Zero tominimize the physical size.

Design DescriptionThe open source EyeTap is modular in nature, these mod-ules fall into three major architectures: mechanical, hard-ware, and software.

Mechanical ArchitectureMechanical architecture is composed of four main compo-nents: the base frame, EyeTap optics, computing appara-tus, and external modules. The base frame is where theother components are mounted, as shown in figure 4. Itconsists of a round piece of plastic with a separate nosepiece. The reasons for this detachable design are: 1. It re-duce the complexity of the 3D printing job. 2. It allows thenose piece to have various sizes to fit comfortably on differ-ent facial structures.

The EyeTap optics consist of the components that are de-scribed in [8], as shown in figure 2. It has: A two sided see-through mirror: this is a cheap alternative to expensivesilver-plated optical beam splitters. Thinner see-throughmirrors are recommended to avoid double image effects. Acamera and an off-the-shelf micro-display with a resolutionof 720*540 pixels.

We have designed spaces to mount external mechanicalmodules to meet the functional needs of various applica-tions. In the current design, there is a reserved spot at thecentre of the main frame for these external modules. Userscan choose to implement various modules such as cam-eras, IMUs (Inertial Measurement Units), and other sen-sors.

Hardware ArchitectureWe propose the use of the Raspberry-Pi system as themain processing component for our design.The choicealigns well with our goal of providing cheap, accessible,open-source solutions to individual developers and hobby-ists. The compact size and low power consumption of theseunits also suits well for a wearable device system. Lastly,the open-source, Linux based operating system, Raspbian,is compatible on any version of the Raspberry Pi, and canbe written into a microUSB card to enable plug-and-playexperience for everyone.

Software ArchitectureAs the Raspbian operating system is entirely open-source,any developer is welcome to develop, add, and interfacethe proposed system with customized components to de-velop individualized applications. By default, we suggest toprovide the developers with stable releases of a wrapper

206

ISWC '17, SEPTEMBER 11–15, 2017, MAUI, HAWAII, USA

application, that acts as a user interface and can be usedto execute the installed applications. The application is pro-grammed to launch automatically on boot-up, and runs infullscreen. The user may navigate through these applica-tions using knobs, buttons or any connected interface, andmay use another button to select them. The operating sys-tem manages the launching and termination of the internalapplications.

Figure 8: Makers wearing theirimplementations of Open SourceEyeTaps.

PrototypesOne particular benefit of 3D printing is that it allows de-signers to go through short iteration. We started with anintegrated design that had most of the components in anenclosure. While this design had most components hidden,it is was very complicated to print, because it required highprecision printers to print it. Thus, making it not accessibleto most at-home makers. Also, this integrated design leftlittle space for other modules to be mounted while havingpoor weight balance near the forehead. In the following iter-ations, we emphasised on design for printability, human fac-tors, and modularity. In our most recent iteration, the weightof the device is reduced by 30% with a more open conceptdesign principle. The evolution of this open source EyeTapdesign is shown in figure 7. The current Open Source Eye-Tap design has been reproduced by multiple makers as auser study, and as a result, over 75% of them were ableto manufacture and assemble this device by themselves.This initiative has gained interest not only from experiencedmakers, but also junior makers and professionals. Somepictures of people wearing their implementation of the Eye-Tap are shown in figure 8.

Future WorkWe are using this design exhibition as a platform for releas-ing our open source augmented reality platform. In the nextstep, we will continue to realize various problems that can

be solved with augmented reality and develop solutions forthem using our platform. This will help us to get an initialapplication base for our platform while also serving as anexample for its use. This will involve detailed documenta-tion of the development process from our base platform toa complete application, added to our application database.We will also conduct more user studies and start pilot pro-grams in schools and universities where students will useour platform to develop solutions to problems in their locality[5][2][19][3]. These pilot programs will be done in variouslocations such as villages in India, China, Silicon Valley andpublic schools in Canada. This will enable us to get a goodspectrum of people to use our platform and show the realpotential of augmented reality.

Software DistributionUpon system release, we propose a forum to be hosted toprovide instructions on releasing applications for develop-ers, to the server. The site will also distribute applicationsand interface releases for download. Lastly, we propose thesite to be used to allow topic discussions and communitysupport.

ConclusionA hardware and software open source wearable augmentedreality platform has been presented. We hope to witnessvarious applications of augmented reality being added tothis platform as time passes. We have thus created tech-nology that will enable the developer, hacker and tinkerercommunity to get their hands on an affordable platform thathelps them use and contribute the the various applicationsof augmented reality [5][2][19][3].

References[1] Mir Adnan Ali et al. “Comparametric HDR (High Dy-

namic Range) imaging for digital eye glass, wearable

207

DESIGN EXHIBITION

cameras, and sousveillance”. In: Technology and So-ciety (ISTAS), 2013 IEEE International Symposiumon. IEEE. 2013, pp. 107–114.

[2] Andrea Bonaccorsi and Cristina Rossi. “Why opensource software can succeed”. In: Research policy32.7 (2003), pp. 1243–1258.

[3] Henry William Chesbrough. Open innovation: Thenew imperative for creating and profiting from tech-nology. Harvard Business Press, 2006.

[4] Jon Evans. “Here comes the wetware”. In: "TechCrunch,https://techcrunch.com/2010/12/04/wetware/" (Dec 4,2010), pp. 27–45.

[5] Rod Furlan. “Build your own google glass [Resources_HandsOn]”. In: IEEE Spectrum 50.1 (2013), pp. 20–21.

[6] Meron Gribetz and Steve Mann. Head Mounted Dis-play Connected with Computational Device and Depth-Sensor with Software Applications and User Inter-face. January 3, 2013.

[7] Steve Mann. “Humanistic computing:" wearcomp" asa new framework and application for intelligent sig-nal processing”. In: Proceedings of the IEEE 86.11(1998), pp. 2123–2151.

[8] Steve Mann. Intelligent image processing. John Wiley& Sons, Inc., 2001.

[9] Steve Mann. “My "augmediated" life: What I’ve learnedfrom 35 years of wearing computerized eyewear”. In:IEEE Spectrum (2013).

[10] Steve Mann. “Phenomenal Augmented Reality: Ad-vancing technology for the future of humanity”. In:IEEE Consumer Electronics (October 2015), cover +92-97.

[11] Steve Mann, James Fung, and Eric Moncrieff. “Eye-tap technology for wireless electronic news gather-ing”. In: ACM SIGMOBILE Mobile Computing andCommunications Review 3.4 (1999), pp. 19–26.

[12] Steve Mann et al. “Blind navigation with a wearablerange camera and vibrotactile helmet”. In: Proceed-ings of the 19th ACM international conference onMultimedia. ACM. 2011, pp. 1325–1328.

[13] Steve Mann et al. “FreeGlass for developers,"haccessibility",and Digital Eye Glass+ Lifeglogging research in a(sur/sous) veillance society”. In: Information Society(i-Society), 2013 International Conference on. IEEE.2013, pp. 48–53.

[14] Steve Mann et al. “Realtime HDR (high dynamicrange) video for eyetap wearable computers, fPGA-based seeing aids, and glasseyes”. In: in Proc. IEEECCECE 2012. Citeseer. 2012.

[15] Glyn Moody. Rebel code: Linux and the open sourcerevolution. Basic Books, 2002.

[16] Sarang Nerkar et al. “Extrapolative Lightspace Methodfor HDR Video Exposure Selection”. In: Multimedia(ISM), 2016 IEEE International Symposium on. IEEE.2016, pp. 397–398.

[17] Pete Scourboutakos et al. “Phenomenologically Aug-mented Reality With New Wearable LED SequentialWave Imprinting Machines”. In: Proceedings of theTenth International Conference on Tangible, Embed-ded, and Embodied Interaction. ACM. 2017, pp. 751–755.

[18] Thad Starner et al. “Augmented reality through wear-able computing”. In: Presence: Teleoperators andVirtual Environments 6.4 (1997), pp. 386–398.

[19] Eric Von Hippel. “Innovation by user communities:Learning from open-source software”. In: MIT Sloanmanagement review 42.4 (2001), p. 82.

[20] Feng Zhou, Henry Been-Lirn Duh, and Mark Billinghurst.“Trends in augmented reality tracking, interaction anddisplay: A review of ten years of ISMAR”. In: Pro-ceedings of the 7th IEEE/ACM International Sympo-sium on Mixed and Augmented Reality. IEEE Com-puter Society. 2008, pp. 193–202.

208

ISWC '17, SEPTEMBER 11–15, 2017, MAUI, HAWAII, USA