1

A Prototype Haptic EBook System to SupportImmersive Remote Reading in a Smart Space

Abu Saleh Md Mahfujur Rahman, Kazi Masudul Alam and Abdulmotaleb El SaddikMultimedia Communications Research LabUniversity of Ottawa, Ottawa, ON, Canada

Email:{kafi@mcrlab, malam@discover, abed@mcrlab}.uottawa.ca

Abstract—Interactive book reading contributes profoundly tothe language development in preschool students. Experimentalresearch showed that shared book-reading between adults andchildren provides children with the opportunity to acquire newvocabulary. Moreover, the time spent reading together providesclear evidence to a child of a parent’s love and care [1].Inspired by such, we propose a remote reading framework inwhich parents or grandparents remotely participate in remotereading sessions. In this framework, we present an intuitiveannotation based approach of hapto-audio-visual interaction withthe traditional digital learning materials. We argue that pictureand haptic modality enhanced book reading accelerates languagedevelopment as students can relate the text with a known visualand tactile references. Hence, in the proposed Haptic E-Booksystem, by integrating the home entertainment system in theuser’s reading experience combined with haptic interfaces weexamine whether such augmentation of modalities influence theuser’s learning behaviour. The proposed Haptic E-Book (HE-Book) system leverages the haptic jacket, haptic arm bandas well as haptic sofa interfaces to provide haptic emotivesignals to the remote story listener in the form of patternedvibrations of the actuators and expresses the learning materialby incorporating image based augmented display in order to paveways for intimate, shared, and immersive reading experience inthe popular ebook platform.

Index Terms—Haptic Book; Annotation; Augmented Render-ing; Haptic Rendering; Shared Reading

I. INTRODUCTION

Book reading sessions are particularly effective for thepreschool children in order to assist them to increase theirvocabularies [2][3]. Senechal et. al. [4] reported that story-book exposure accounted for unique variance in preschoolchildrens expressive and receptive vocabulary after controllingfor parents education, parents own level of literacy, andchildrens analytic intelligence. Experimental research showedthat shared book-reading between adults and children provideschildren with the opportunity to acquire new vocabulary. Theidea that storybook reading promotes language developmentis supported by correlational, experimental, and interventionstudies. In general, children loves story book reading sessions,specially when they hear it from their grand parents or parents[5]. The time spent reading together provides clear evidenceto a child of a their love and care [1]. However, it is notalways possible to share the reading experience with the childor a grandchild due to separate geographical locations anddistances. In this paper, we address a remote reading systemthat engages the reader and the listener to participate in an

entertaining reading experience and strengthen their emotionalbonds. We propose a remote reading framework in whichparents or grandparents remotely participate in remote readingsessions in this paper.

Our ways of readings are continuously moulded with theadvent of new technological innovations, devices and plat-forms. For example, e-book is continuously challenging theexistence of print book and at times ebooks are replacingthe former in schools [6]. In our daily life, we are gettingmore and more explored to electronic materials than printedbooks as now-a-days most of the advanced hand held mobiledevices provide e-book reading facilities [7]. Also, today’s usercommunity is widely accepting haptic interfaces; the effectof tactile feedbacks from the reading materials has startedto form a new genre of research interest. Moreover, picturebook reading accelerates language development [8][9][10] aslearners can relate the text with a known visual reference.Colourful illustrations provided the opportunity for narratingthe story without complete reliance on the text [4]. Hapticfeedback and visual images relating to the reading contentalso assist the learner to memorize words by associating theirviewing and emotional experiences with the learned words[11]. Hence, in the remote reading system, in order to leveragethe use of such modalities we incorporate intuitive approachof annotation based hapto-audio-visual interaction with thetraditional digital learning materials. A complete overview ofthe components of the remote reading system that supportslearning material centric haptic and visual content renderingis depicted in Figure 1.

Our proposed remote Haptic E-Book (HE-Book) is a specialtype of e-book reader, which is capable of delivering contentrelated visual images as well as vibrotactile feedbacks to user’sremote reading experience. Specially, in a home entertainmentscenario, where the grandparents read the content remotely byusing their customized eBook clients and the children view thevisual images and receive haptic emotional feedbacks that arerelated to the learning content on their television screen andwearable haptic jackets respectively [12]. We incorporate thehaptic stimulations defined in our previous work [12]. More-over, the haptic sofa1 provides content dependent vibrotactilefeedbacks such as boat or train movements.

In order to deliver such experiences, we present a pro-totype remote HE-Book system exploring the suitability of

1Haptic sofa, http://d-box.com/en/home-theatre/d-box-ready-seating/

978-1-4577-0499-4/11/$26.00 ©2011 IEEE

2

Fig. 1. High level architecture of the proposed HE-Book system

haptic feedbacks and multimedia contents in various e-bookplatforms. Our contribution to this research article is twofolds. First of all, we propose the annotation based haptic-audio-visual remote feedback integration by authoring hapticas well as multimedia data in the traditional digital textcontents. Secondly, we present a generalized framework tosupport remote reading on that and illustrate the variousremote interaction mechanisms. We introduce support forboth individual and remote group reading scenarios whereubiquitous home multimedia devices are incorporated withinthe reading activities.

The remainder of this paper is organized as follows. In thebeginning, in Section II describe the multimedia annotationscheme. Further in Section III, we illustrate two key compo-nents that allows the remote rendering communication in oursystem. Next in Section IV we present various components ofour system that facilitates the annotation based hapto-audio-visual delivery from digital reading materials. We concludethe paper in Section VI and state some possible future workdirections.

II. MULTIMEDIA ANNOTATION SCHEME

Fig. 2. Annotation editor for authoring hapto-audio-visual materials.

We develop an annotation tool to author the image andhaptic feedbacks for the ebook contents. The annotation toolis shown in Figure 2. By using the developed annotationscheme an author selects a paragraph of a given page of ane-book and tags those by using various haptic and multimediaproperties [13]. For multimedia:image annotation steps, after

selecting the texts of the ebook, the annotator is provided witha list of possible images from the available image providersbased on web search results. The images are depicted in animage container for easy selection. Later, an author selects theimages that describes the selected texts more appropriately.The selected images are stored in the annotation file. Incase of haptic annotation, we currently support three hapticdelivery devices. They are haptic jacket, haptic arm-band, andhaptic sofa. For each haptic device we create custom vibrationpatterns, use appropriate names for the patterns and enlistthem in the authoring tool. The annotation XML file (Fig.3) contains image list, vibration types for selected vibrationdevices for each authored ebook contents.

We keep the annotation file separated from the actual ebookcontents. The advantage of separate XML based annotationfile is that a HE-Book user could decide to either use themultimedia+haptic extension or avoid it. A possible businessmodel for the separate XML file can be a HE-book XMLstore from where user can consume annotation file for aspecific e-book. For the annotated paragraph of an e-book,we have created an XML element Page which has attributeID that denoted the unique number of the page. Every Pageelement is divided in Para which also has ID that denoted theparagraph no. Granularity level of our annotation ends at Parai.e. vibrotactile feedback and multimedia playback start andend in a paragraph level, which we term as scene or paragraphbased annotation. Under the Para we have elements Haptic andImage for our prototype purpose which is extended to supportinternal, external audio, video too. Haptic is described usingDevice Type, Pattern, Repeat and Delay and Image annotationcontains Src, URL attributes. Motivated by our past work [12]we annotas the e-book content with Touch, Fun, Hug, Tickle,Kiss, Poke type haptic feedbacks into our HE-Book system.Annotation Retriever is responsible of searching the annotationfile when it is necessary and retrieves the requested XMLblock. In a nutshell this is an XML parser. For a requestof {Page, Para} pair, this XML parser returns correspondingHaptic, Image blocks from the XML file.

III. REMOTE EBOOK READER SYSTEM

Here we describe the reader server and the listener serverand how these server handles ebook interactions and provide

3

hapto-audio-visual feedbacks to the listener smart devices.Two components of the system are the reader and listenermodules. In a typical reading session, the reader moduleprovides the reader to control the pace and content of thereading. While the listener module renders the reading contentthrough audio, visual and haptic channels. When the computervoice is reading aloud the book content, the listeners view therelated pictures on a television and receive haptic feedbacksin their connected haptic devices.

A. Reader Interaction Controller

Interaction Controller module that resides in the Readerserver plays the central role to organize and synchronizenecessary work flow in the haptic e-book reading system.The module polls to acquire user touch inputs after certaininterval. As soon as a touch interaction is performed theReader Interaction Controller coordinates the screen locationbased paragraph identification of the e-book document. TheAnnotation Manager module takes the page ID and the para-graph keys to retrieve the haptic and visual data associatedwith the paragraph and returns the xml data to the Interac-tion Controller. These xml content along with the page andparagraph keys are further sent to the Listener module.

The haptic e-book system can work in a touch based mobiledevice. The user can flip pages and scroll the paragraph ofthe pages through the touch based interaction. We use thestandard touch SDK is used to obtain the screen coordinatesby using the GUI Interaction Listener. The Listener uses theParagraph Locator component in order to match the touchedscreen coordinate with the displayed e-book page content todeduce the paragraph that the user is currently pointing at.The e-book XML reader software continuously monitors thetouched screen coordinate interaction. As soon as the usertouches a paragraph of the page that has been previouslyannotated, the packaged xml annotation keys are sent promptlyto the Listener module.

Fig. 3. XML based annotation file for the e-book document.

B. Listener Ambient Media Service

The Listener module consists of the Ambient Media Ser-vices that connect with haptic devices, audio system, and videosystem. The Listener module listens to the remote Interaction

Controller module and provides the needed haptic, audio,visual signals. The implemented Listener module is connectedthrough peer-to-peer based architecture and transfers messagesin a socket port. In the beginning the Reader module sendsthe Listener module the name of the books and transfers thexml annotation file. The xml file is parsed and loaded intothe memory in a custom data structure list. In order to accessthe individual elements of the list the Listener module requirepage and paragraph keys. At all times the Listener modulewaits for the page and paragraph keys that it receives fromthe Reader module. As the Listener module accepts the keys,it locates the annotation element from its loaded data structure.

In the data structure list the Listener module locates theneeded haptic-audio-visual data. These data are further sent tothe Multimedia Rendering Manager. The Multimedia Managermodule is responsible for producing the signals to render thehaptic, audio, visual devices. The listener user then obtainsthe assigned haptic, auditory and visual feedback associatedwith the learning content. The Listener module also polls thesurrounding active devices. If any media device is discoveredthen paris with the devices and performs preliminary messagecommunications in order to maintain the media synchroniza-tion. Listener module also checks the status of a device inorder to ensure whether the device type is a standalone orintermediate. Standalone device is itself capable of continuingmedia streaming but for a setup such as group entertainmentan intermediate computer device play the role as mediatorcontroller. In addition to coordinating the aforementioned taskthe Listener module makes sure that the system modules arenot blocking or throttling the operation of the other modules.The Listener module employs carrier-sensing algorithm todetermine the active/idle states of the other modules.

IV. MULTIMEDIA DELIVERY SCHEME

The multimedia manager module resides in the mobiledevice client and handles the hapto-audio-visual rendering.The rendering schemes are described as the following.

A. Haptic Renderer

Based on the page and paragraph information of TouchBased GUI, Interaction Controller gets haptic descriptionfrom the Annotation Retriever sub-module and generates thedescribed haptic signal for the targeted haptic device. HapticSignal Generator plays an important role for device specifichaptic signal generation. As the target haptic device can beheterogeneous we considered device specific configuration.In our system architecture, we have considered Bluetoothas a method of communication between our system and thecorresponding haptic devices. Such communication can alsobe extended to other possible personal network communicationmethods based on the device support. In our prototype system,we have used a Bluetooth enabled haptic jacket [12], a D-Boxhaptic sofa [11], and a Bluetooth enabled haptic arm-band[14][15].

From our previous work [12] we have selected three basicemotions {Love, Joy, Fear} based on the haptic devices avail-able for our prototype system. Above described emotions are

4

tagged to the e-book content semi-automatically as describedearlier. We also have annotated real life scenarios such asriding bike, driving car, airplane, travelling by boat, sea stormtide effects, holding hands, tickle, hug, poke, touch. We definethese haptic feedbacks empirically in the following.

• Touch: In order to mimic touch haptic interaction wehave incorporated funnelling illusion based actuator pat-tern generation. The patterned vibrations of the set ofactuators on the human skin produce the touch sensations.

• Poke: Haptic poke incorporates the similar patterned vi-brations as the touch. However, the funnelling parametersfor the motors are reduced and the vibration strength ishigher in this type of haptic simulations.

• Tickle: Through the haptic jacket interface, we empir-ically have positioned the actuator motors around thebelly, armpit and neck area and leveraged a set ofpatterned touch sensations in order to generate ticklehaptic feedback.

• Holding Hands: By using the haptic armband we haveachieved the holding hand haptic feedback. In this typeof haptic rendering the haptic touch is focused on thearmband area.

• Sea Storm, Travelling in a Boat/Airplane/Car, Bike Rid-ing: The haptic sofa is responsible for the generationof such feedbacks. The sea storm haptic feedback is aspecial type of boat riding haptic feedback. In order togenerate boat riding haptic sensation the D-Box interfacegenerates wave of signals to modulate the actuatorsaccordingly. In car travelling haptic feedback we modifiedthe boat travelling feedback and enhanced it with frequentvibrations. In order to simulate bike-riding experience,the haptic sofa reduces the actuator vibrations, however,increases the force of the actuators in the process.

• Fear, Joy, Love: We again capitalize the haptic touchand haptic poke feedbacks in order to generate theseemotional feedbacks. In case of fear haptic rendering,we enable the periodic patterned actuator touch sensationalong the backbone. The butterfly joy effect is producedby incorporating a series of mild-poke and mild touchsensation around the stomach area. Whereas the loveeffect is produced by enhancing the haptic touch aroundthe left-chest area. The actuator motors were placedaround these said areas with varying intensity beforehand.

After empirically defining the haptic sensations we use thosein the annotation steps. The visual image contents and hapticsensation type, frequency and rendering types are all definedin the xml annotation file. The Listener module dynamicallyrenders these haptic vibrations on the available haptic devicesin the smart home environment.

B. Image Rendering Subsystem

In this section we describe the text-to-speech interactionmodalities, 2D image rendering approach and multimediadelivery scheme to the home entertainment system into moredetail. The Imager Viewer incorporates augmented visualfeedback on the television screen so that a learner can ’see’images that are related to the learning content. The images

are statically defined during the annotation steps for eachparagraphs of a book. A semi-automated process allows theusage of the vast image database and makes those availableduring the reading process. Seeing an image of a snow stormwhile reading about it can help the user improve his/herreading and understanding abilities in order to help practiceeffective learning behaviour [16]. Additionally, the augmentedvisual display of the learning material can make the intuitivelearning process more interactive and entertaining [17].

Home multimedia system is suited for group listeningscenario. When the remote reader interacts with the annotatedeBook content using his/her eBook client then the remotelisteners can enjoy the content related information by utilizingthe smart devices surrounding them such as Haptic sofa, hapticjacket, haptic arm-band, stereo sound system, HD televisionetc. For example, a father from remote travel location reads atravel to Venice city using his Ebook client. The reader’s wifeand young kids are listener users. Both mom and the kid arewearing haptic jackets, sitting on a haptic sofa and in frontof them there is a HD Samsung LCD 46” 1080p TV. Whilereading the book, a remote user touches annotated part of thecontent and consecutively a list of images are processed bythe Listener module and queued to be displayed on the TV.At the same time, wearable and surrounding haptic devicesof the listener users start to play the command that is sent toit. In our prototype system, haptic signalling and streaming ishandled by an intermediate computer, which is connected tothe eBook client by using a personal communication network.TTS support of the Listener module makes it possible fora listener users to listen the reading content in a computervoice that is rendered in a computer connected audio playbackdevice.

Fig. 4. HE-Book component architecture

V. IMPLEMENTATION DETAILS

In this section, we present the implementation details of thehaptic e-book system. The implemented system componentsare shown in Figure 4. One of our prototypes was developedfor desktop e-book readers by using Netbeans 6.5 IDE andthe primary language was JAVA. In order to develop the e-book reader, we locally build the ICEPdf 2 open source JAVAPDF viewer. We have used a Dell touch screen monitor forour system and added the touch processing part to the ICEPdf.We also have added XML retrieving facilities to the ICEPdf

2Open Source Java PDF, http://www.icepdf.org

5

for annotation searching. For serial port based Bluetoothcommunication we have used GPL based BlueCove3 librarywhich is very handy for J2SE based Bluetooth communication.Our prototype system communicates with a haptic jacket whenany annotated e-book part is touched by the user. A Bluetoothdevice was connected with the PCs USB port, which wasvirtually configured with the COM port so that the Bluetoothdevice can send signals to the haptic jacket. Our desktop basedtest prototype was adequately responsive in a standard Pentiumdual core 32-bit machine with 2 GB systems RAM. In orderto annotate an e-book we have taken PDF as example anddeveloped a PDF annotator modifying the ICEPdf that wehave used as viewer also. We have added tagging windowto the system where we have considered various taggingoptions. An operator can select a paragraph of a PDF file andtag it using our prototype editor. For each tagging an XMLblock is created which is edited by the human operator forvarious haptic attributes. Later this XML file is stored to befurther used by the ICEPdf viewer while reading annotatede-book. We used RFCOMM based the Bluetooth Serial PortProfile communication with the end haptic devices. Our endbluetooth devices were haptic jacket and haptic armband. Thecommand format was suitable for sending/receiving bluetoothsignals to the haptic devices Parani ESD200 Bluetooth kit.For text-to-speech version of the mobile system we have usedjava.speech package under JSR 13 Java Speech API.

We achieved remote communication by using client-serverbased message delivery mechanisms. The designed the remoteReader module as a client to the Listener module. For ourhome multimedia prototype, we have combined above de-scribed two systems in one system. When a reading eventis initiated from the Reader module a specific commandis transmitted to the Listener server system. Correspondingdesktop system receives the command and distributes variousmedia to related devices. For the home multimedia version, westream stream annotated pictures from the Listener system. Incase of image streaming, picture data is received in the desktopend and displayed to the Samsung LCD 46” 1080p TV.

VI. CONCLUSION

Human computer interaction research has gained a pacewith the advent of haptic devices. Our research goal is toexplore the possibilities and opportunities this modality couldbring to the remote shared reading experience. In this regard,we introduced an intuitive remote e-book reader system.The system works by using Reader and Listener interactionmodules. The interaction of remote reader is capture by theReader module and selected reading content is sent to theListener module. The Listener module looks up the annotationof the current reading content and further delivers hapto-audio-visual feedbacks to the listener users.

3BlueCove, http://www.bluecove.org

Current system can meaningfully synchronize all media forparallel devices to form a better orchestra of learning andentertainment. The remote reading system can benefit thelearning experience of the remote users and improve theirvocabulary and language learning capabilities. In future wewant to perform a detailed usability study of the proposedapproach to measure the impact of remote delivery of hapto-audio-visual materials in the knowledge acquisition process.

REFERENCES

[1] M. Fox and J. Horacek, Reading magic: Why reading aloud to ourchildren will change their lives forever. Mariner Books, 2008.

[2] C. J. Lonigan and G. J. Whitehurst, “Examination of the relative efficacyof parent and teacher involvement in a shared-reading intervention forpreschool children from low-income back- grounds,” Early ChildhoodResearch Quarterly, vol. 13, no. 2, pp. 263–290, 1998.

[3] A. Hargrave and M. Senechal, “A book reading intervention withpreschool children who have limited vocabularies: The benefits of regu-lar reading and dialogic reading,” Early Childhood Research Quarterly,vol. 15, no. 1, pp. 75–90, 2000.

[4] M. Senechal and J. LeFevre, “Long-term consequences of early homeliteracy experiences.” in The annual meeting of the Canadian Psycho-logical Conference. Edmonton, Canada, 1998.

[5] H. Janes and H. Kermani, “Caregivers’ story reading to young chil-dren in family literacy programs: Pleasure or punishment?” Journal ofAdolescent & Adult Literacy, vol. 44, no. 5, pp. 458–466, 2001.

[6] A. Mangent, “Hypertext fiction reading: haptics and immersion,” Re-search in Reading, vol. 31, no. 4, pp. 404 –419, nov. 2008.

[7] B. Schilit, M. Price, G. Golovchinsky, K. Tanaka, and C. Marshall, “Thereading appliance revolution,” Computer, vol. 32, no. 1, pp. 65 –73, jan.1999.

[8] D. H. Arnold, C. J. Lonigan, G. J. Whitehurst, and J. N. Epstein, “Accel-erating language development through picture-book reading: Replicationand extension to a videotape training format.” Journal of EducationalPsychology, vol. 86, pp. 235–243, 1994.

[9] L. Dunn and L. Dunn, Peabody Picture Vocabulary Test–Revised. CirclePines, MN: American Guidance Services, 1981.

[10] M. F. Gardner, Expressive one-word picture vocabulary test–Revised.Novato, CA: Academic Therapy Publications, 1990.

[11] A. S. M. M. Rahman, K. M. Alam, and A. El Saddik, “Augmentedhe-book: A multimedia based extension to support immersive readingexperience,” in Autonomous and Intelligent Systems (AIS), 2011 Inter-national Conference on, To be published, 2011.

[12] A. S. M. M. Rahman, S. A. Hossain, and A. El Saddik, “Bridging thegap between virtual and real world by bringing an interpersonal hapticcommunication system in second life,” in IEEE International Symposiumon Multimedia, Taiwan, Nov 2010.

[13] A. S. M. M. Rahman, J. Cha, and A. El Saddik, “Authoring edutainmentcontent through video annotations and 3d model augmentation,” in IEEEInternational Conference on VECIMS, Hong Kong, China, May 2009,pp. 370–374.

[14] A. S. M. M. Rahman and A. El Saddik, “hkiss: Real world based hapticinteraction with virtual 3d avatars,” in IEEE International Conferenceon Multimedia and Expo, HFM2011, Barcelona, Spain. 2011.

[15] A. S. M. M. Rahman, M. Eid, and A. El Saddik, “Kissme: Bringingvirtual events to the real world,” in Virtual Environments, Human-Computer Interfaces and Measurement Systems, VECIMS 2008, Istanbul,Turkey, July 2008, pp. 102–105.

[16] E. Dale, Audio-Visual Methods in Teaching. Rinehart and Winstor,1969.

[17] M. Billinghurst, H. Kato, and I. Poupyrev, “The magicbook - movingseamlessly between reality and virtuality,” Computer Graphics andApplications, IEEE, vol. 21, no. 3, pp. 6 – 8, 2001.