SMAART - dcm.uhcl.edudcm.uhcl.edu/caps11g5/tech.docx · Web viewACKNOWLEDGEMENT. We take an...

SMAARTSophisticated Mobile Application using Augmented Reality

Technology

TECHNICAL REPORT

SPRING 2011

Instructor:

Dr.Kwok-Bun YueMentor:

Mr.Dilhar De SilvaProject Team Members:

Nithin Kumar DuddeManoj Gidda

Shivani KovvuriSrikanth Inapurapu

Srikanth Majji (Team Leader)Report Date:

29 April 2011

Sophisticated Mobile Application using Augmented Reality Technology

ACKNOWLEDGEMENT

We take an immense pleasure in expressing our sincere thanks to our instructor, Dr. Kwok Bun

Yue, for providing us with valuable feedback throughout the capstone project. We really

appreciate his advice, support, and feedback.

We are very much grateful to our mentor, Mr. Dilhar De Silva, for his inspiring guidance and

advice throughout our project. Mr. Dilhar De Silva has been the inspiration and was always up

front to motivate and encourage us for bringing out our project successfully. We would like to

thank him for his excellent technical expertise, guidance, support, and time.

Finally we extend our thanks to each of our families, who gave us encouragement whenever we

had a tough time.

Capstone Spring 2011 2 | P a g e


ABSTRACT

In today’s world, mobile applications are triggering a fundamental shift in the way people

experience computing and use mobile. The explosive growth in smartphones in the last three

years facilitated the development of hundreds of thousands of mobile applications that promised

to fundamentally change how people use the Web. Our mentor, Mr. Dilhar De Silva, CEO of

AtLinks Communications, was interested in developing an android mobile prototype application

using augmented reality technology.

The overall goal of the project is to develop an Android-based augmented reality (AR) mobile

application through which images and text can be placed on top of the objects of the phone

camera view based on the geo-location of the objects detected by the phone. For example, using

this application, administrative user can place the schedule of the event at a particular geo-code

(latitude and longitude) on a server by using a web application and the mobile users can open the

mobile application on android phone to view the augmented image of schedule overlying on

camera’s view. Mobile user can also download the schedule into their mobile phones by

providing their login credentials.

To provide the necessary AR functionality, our team developed and used a metric to study about

AR API’s and selected QUALCOMM API for pattern based augmentation, and WIKITUDE API

for geo-code based augmentation.

This project was carried out in accordance with Rational Unified Process (RUP), using Scrum

process. The process went iteratively and incrementally delivering sprints and finally delivering

the end product. The project will be documented using UML 2.0.



CONTENTSACKNOWLEDGEMENT 2

ABSTRACT 3

1. INTRODUCTION 5

1.1 Purpose 5

1.2 Overview 5

2. SOFTWARE DEVELOPMENT MODEL 8

3. DESIGN AND IMPLEMENTATION 10

3.1 Architecture diagram: 10

4. TECHNICAL DETAILS 13

5. Technical Challenges and Lessons Learnt 14

5.1 Selection of Android Mobile phone 14

5.2 Selection of Augmented Reality Browser/Engine 14

6. CONCLUSION 15

7. FUTURE WORK 15

8. REFERENCES 16

9. APPENDICES 17

Appendix A: Project Management and Team Information 17

Appendix B: Major Tasks and Contributions 18

Appendix C: Software Requirements Specification 19

Appendix D: Database Design Schema 23

Appendix E: Use case and Sequence Diagrams 26

Appendix F: Screenshots of Client Side and Server Side 36



1. INTRODUCTION

1.1 Purpose

The goal of our capstone project is to develop an Android-based augmented reality (AR) mobile

application through which images and text can be placed on top of the objects of the phone

camera view based on the geo-location of the objects detected by the phone. It will also develop

a server-side web application where users can place graphical image at a specific geocode

(location) using Google Maps.

1.2 Overview

In today’s business and educational world, every industry and educational institutions wants to

share any specific information at lightning speed. This may be a part of manufacturing industries

promoting their new product, or shopping malls and restaurants promoting their new discount

offers, or educational institutions announcing on-campus job vacancies and any special events

etc. In the existing system, manufacturing industries does these promotions by displaying the

new product details on big hoardings or by advertising it on Television. Shopping malls and

restaurants promote their discount offers and by adding pamphlets in the daily news paper.

Students of university have an only option of reaching the special event place to obtain the

schedule of it.

However the existing system of sharing the specific information to people has several drawbacks

1) Cost factor in case of advertising on TV and displaying large hoardings.

2) Pamphlets outmoded - Majority of the news paper readers are more interested on reading

news rather than the pamphlets added in it.



3) Students would not know the schedule of an event unless they reach the place and also

printing schedule brochures would add an extra budget to the event.

In today’s world of technology, internet is one of the most important technological

breakthroughs of the 20th century along with the mobile computing. “According to the results of

a recent online survey conducted by IBM, television sets are losing ground to the internet when it

comes to personal leisure time”[6]. These kinds of promotions and advertising based on the

existing systems mentioned above are reaching to very less people.

As the growth of internet and mobile users are growing day by day, all the mobile carrier

services are offering smart phones to them at low prices possible with a contract based. Our

mentor Mr. Dilhar De Silva wants us to build a mobile application on android using augmented

reality technology to overcome the above drawbacks by targeting all the android mobile users as

an initiative step.

SMAART, an android-based augmented reality technology application makes android mobile

phone users life easier in finding the information they looking for. Rather going to the places in

person, an android mobile phone owner can download an augmented reality application/software

that uses the phone's camera and GPS capabilities to gather information about the surrounding

area. For example, if the user selects restaurants from the categories available, it shows

information (order menu and discount offers) about restaurants in the area, overlaying this

information on the phone's screen. You can even point the phone at a building, and it will tell

you if any companies in that building are hiring by displaying a text image on your mobile


http://electronics.howstuffworks.com/gadgets/travel/gps.htm

http://www.marketwire.com/mw/release.do?id=762949


phone. Aiming the camera towards the educational institution would fetch the mobile user with

an event schedule overlying on the phone’s screen.

The server side web application shall have a web based interface which uses Google Maps API

to aid administrative users to place the images at specific geo-code locations. Using sensor

services such as GPS, the mobile application shall get the current geo-code (latitude and

Longitude).The current geo-code (latitude and Longitude) of the device along with user selection

of categories and the radius (distance user wants to find places within) is sent to server to figure

out if any images have been placed in the virtual world. The images that are augmented are

displayed through the camera of the device. The mobile users then should be able to download

and store the image augmented onto the device.



2. SOFTWARE DEVELOPMENT MODEL

We used Rational Unified Process (RUP) as the software development model. It provides a

disciplined approach to assign tasks and responsibilities within a development organization. The

goal of RUP is to ensure the production of high-quality software that meets the needs of its end-

users, within a predictable schedule and budget.

Fig 1: RUP phases

RUP has a project lifecycle that consists four phases.

Inception: In the inception phase, the team met with Mr. Dilhar De Silva and gathered

all the requirements. One of the requirements of our project is researching on different

AR browsers and selecting the best open source browser that suites our project. We have

selected Qualcomm API for pattern recognition and Wikitude API for geo-location. The

team developed a basic use case diagram depending upon the requirements. Our team



also made research on different kinds of android mobile phones and chose the Dell Streak

for our project. We have also learnt about the Android SDK by dividing it into sections

and later explained it each other. The team worked for almost three weeks in the

inception phase.

Elaboration: In the elaboration phase, we have revised the basic use case diagram. The

team developed sequence diagrams as well. The team also designed a high level

architecture diagram showing the flow of communication between the client and server.

The team worked for four weeks in the elaboration phase.

Construction: The construction phase was almost seven weeks. The team learnt new

technologies like Android SDK, Qualcomm API, and Wikitude API. We have divided the

team into two groups. One group working on server side and other group working on the

client side. As we have divided the work, we were able to work independently and

complete the project on time.

Transition: During the transition phase, the team performed some tests to verify the

project requirements. The prototype was demonstrated to Mr. Dilhar De Silva to validate

that the project met the end users’ expectations. The team worked for two weeks on

transition phase.



3. DESIGN AND IMPLEMENTATION

3.1 Architecture diagram:

The architecture diagram shows the working of the project in detail. Images uploaded on to the

server by authorized user will be stored on the database as shown in the diagram below. Client

side mobile application receives the images from the server to augment it on the mobiles camera.

The technical details of the project are explained in next section.



The communications between the client, server and database is explained

Communication between Client & Server:

1. On the Android phone, the user shall select the category to view his required choice of

images or text overlying on camera’s view by clicking on categories menu button.

2. A mobile user shall set the radius (maximum distance to consider from the current geo-

code) to display the requested category of images with in the distance of radius.

3. The current geo-code, along with selected categories and radius is sent to the server as a

HTTP request.

4. The client application receives all the images of user-selected categories within the user-

defined radius as an XML file from the server.

5. Once the augmented images are displayed on the camera’s view, user can select an image

for download, which in turn requests for mobile user login credentials.

6. Once the mobile user provides the login information, server authenticates the user and

then he/she would be able to download the image on to the device on a successful login.

Communication between Server & Database:

1. When authorized users provide login credentials to logon to web application, the server

checks with the database to authenticate them.

2. Whenever an authorized user adds an image along with the image name and description

on the server side application at a specific geo-code, it is stored in to the database.



3. When a HTTP request with geo-code, informational categories and radius are received

from client application, the server checks with the database to find images of the

categories within a certain radius.

4. TECHNICAL DETAILS

This section explains the various technologies we used to develop this application.

Mobile Application (Client Side):

The Client side application was implemented using Java on Android 2.2 platform. We have used

WIKITUDE API for augmenting images on camera view based on geo-code. We have used

QUALCOMM API for augmenting image on camera view based on pattern recognition. We

have also used Google Maps API on client side and placed markers on server-specific geo-codes.

Web Application (Server Side):

The Server side web application was implemented using Java server pages for user interface and

MySql for the database. We have used Google Maps API to place the marker and add images

with description at user-specific geo-code.



5. TECHNICAL CHALLENGES AND LESSONS LEARNT

5.1 Selection of Android Mobile phone

The first challenge the team had was, determining what Android phone to use for the project.

The team gathered all the requirements of the project to determine the required functionalities of

android phone. The team determined that, an android phone supporting a Qualcomm API and

GPS would be the most reliable for the application. The team has done good research and

selected some android phones which have satisfied the requirements criteria. By comparing the

chosen android phones based on certain metrics, the team has decided Dell Streak 5 will satisfy

the requirements of the project. The Dell Streak 5 has a built-in Qualcomm chip and GPS which

would be used both for pattern recognition and geo-code based augmentation. Both mentor and

instructor have agreed and recommended our selection of android phone.

5.2 Selection of Augmented Reality Browser/Engine

Another challenge the team had was determining the best AR browser/engine for the project. AR

browser/engine is used to augment images or text on top of the objects of camera’s view in our

project. The team has done research on it and come across many AR browsers/engines. Based on the

metrics, the team has compared all the AR browsers/engines and found Qualcomm API is best

suitable for a pattern-based augmentation. Qualcomm API is an open sourced API and does not



augment images based on the geo-code; it augments images based on pattern recognition. The team

has determined Wikitude API as the best AR browser in augmenting images based on the geocode.

Wikitude API is an open sourced browser and it is very well documented.

6. CONCLUSION

Our team was successful in completing all the requirements that were specified, even though the

concept of augmented reality was entirely new for all our team members. In the process of

completing this project, we were able to learn various new technologies. Learning all these new

technologies and providing a working prototype in the short aggressive schedule was

challenging. The team enjoyed working with the mentors and getting the opportunity to learn

from this project. Overall, we had good experience working on the capstone project.

7. FUTURE WORK

In future, performance of the application (SMAART) can be improved using WIKITUDE API to

augment images on camera based on geo-location. SMAART, an augmented reality based

android application has also been able to augment images on camera using a basic pattern

recognition with the help of Qualcomm API. There is lot of scope to enhance on pattern

recognition based augmentation in this application.



8. REFERENCES

[1]. IBM Rational Software. "Rational Unified Process : Best Practices for Software

Development Teams." IBM DeveloperWorks: Rational. January 10, 2003.

http://www.ibm.com/developerworks/rational/library/content/03July/

1000/1251/1251_bestpractices_TP026B.pdf

[2]. Benismobile (November 23, 2010). Comparing AR browsers. Retrieved from

http://mobilegeo.wordpress.com/2010/11/23/comparing-ar-browsers/ for comparing different AR

browsers.

[3]. Learning about Qualcomm API

https://ar.qualcomm.com/qdevnet/developer_guide

[4]. Information about Android fundamentals

http://developer.android.com/guide/basics/what-is-android.html

[5]. Basic knowledge about Augmented Reality

http://en.wikipedia.org/wiki/Augmented_reality

[6]. Survey between Television and Internet

http://www.bit-tech.net/news/2007/08/23/tv_vs_the_internet_internet_wins/1


http://www.bit-tech.net/news/2007/08/23/tv_vs_the_internet_internet_wins/1

http://en.wikipedia.org/wiki/Augmented_reality

http://developer.android.com/guide/basics/what-is-android.html

https://ar.qualcomm.com/qdevnet/developer_guide

http://mobilegeo.wordpress.com/2010/11/23/comparing-ar-browsers/

http://www.ibm.com/developerworks/rational/library/content/03July/1000/1251/1251_bestpractices_TP026B.pdf%20

http://www.ibm.com/developerworks/rational/library/content/03July/1000/1251/1251_bestpractices_TP026B.pdf%20


9. APPENDICES

Appendix A: Project Management and Team Information

Srikanth Majji: Team Leader, Research Analyst, Developer, Database Administrator

Srikanth Inapurapu: Software Designer, Developer

Shivani Kovvuri: Software Designer, Technical writer, Webmaster

Nithin Kumar Dudde: Software Designer, Developer, Webmaster

Manoj Gidda: Research Analyst, Technical writer

Project schedule:

Week Date PhaseWeek 1 01/20/2011

Inception phaseWeek 2 01/25/2011Week 3 02/01/2011Week 4 02/08/2011

Elaboration phaseWeek 5 02/15/2011Week 6 02/22/2011Week 7 03/01/2011

Construction phaseWeek 8 03/08/2011Week 9 03/15/2011Week 10 03/22/2011Week 11 03/29/2011Week 12 04/05/2011

Transition phaseWeek 13 04/12/2011Week 14 04/19/2011Week 15 04/26/2011Capstone Spring 2011 16 | P a g e


Appendix B: Major Tasks and Contributions

Tasks SrikanthMajji (%)

SrikanthInapurapu

(%)

Shivani Kovvuri

(%)

Nithin Kumar Dudde

(%)

Manoj Gidda (%)

Research & Analysis on different AR browsers 20 20 20 20 20

Documentation on Abstract, Agenda &

Meeting Minutes20 15 35 15 15

Modeling Use cases and sequence Diagrams 15 15 15 35 20

Software Requirements Specification(SRS) 15 20 20 15 30

Website Creation, Maintenance 15 15 20 35 15

UI Design 15 35 20 20 15

Database Design 35 10 10 20 25

Database connectivity with UI 25 25 15 15 20

Technical Report 15 20 35 15 15



Appendix C: Software Requirements Specification

Client side:

Open the Application

User shall start the application by clicking on the shortcut key of the application in the

Android phone.

System shall retrieve the current location of the user using GPS.

Choose a category

When the user selects the category from menu, the category list shall be retrieved from the

server and displayed to the mobile user.

User shall select any one of the categories that he/she is interested in from a number of

categories available. (For example categories may include Restaurant, Motel, School, Coffee

shops etc.)

System shall retrieve all the markers belonging to that particular category by connecting to

the server.

System shall display all these retrieved markers to the mobile user.

Set Radius

User shall set the radius to a particular range. The default radius is set to 10 miles and the

maximum radius is set to 60 miles.



System shall retrieve all the markers within that particular range.

System shall display all these markers to the mobile user.

Open App Camera

User shall select the camera option once he/she is done selecting the categories and setting

the radius.

User shall select the augmentation type. The two types of augmentation are using geolocation

and using pattern recognition.

Open Map View

User shall open the map view option once he/she is done selecting the categories and setting

the radius. The map shall display markers in the map belonging to that particular category

within the selected radius range.

User shall select the augmentation type. The two types of augmentation are using geolocation

and using pattern recognition.

Download images

User shall download and store the image selected into the Android phone memory.

When the user shall download an image, the system asks the user for login information.

Open images

User shall open a stored image from the Android phone memory.

User shall need a JPEG viewer to open the stored images.



Server side:

Login to web server

System shall allow the user to enter a valid mobile number and password.

System shall successfully login and display “login successful” when authenticated.

System shall prompt an error message when an invalid user logs in or when the user does not

enter his mobile number and password.

User Registration

User shall enter mobile number which is 10 digits.

User shall enter a password which should be a minimum of 8 characters.

User shall enter first name.

User shall enter last name.

User shall enter email.

User shall enter mobile model number.

User shall click on “submit” after entering all the details.

If the mobile number already exists, the user shall be displayed an error message that the

“mobile number already exists”.

Pick geo-location

User shall select a geo-location from Google Maps.



System shall display the latitude and longitude of the geo-location selected.

Add image at geo-location

User shall add an image to the geo-location he/she has selected.

Create category

User shall create a new category and add an image to that particular category.

Categorize image

User shall categorize the image to any one of the categories that is been added to a particular

geo-location.

User shall create a new category if he/she wants to.

Update image

User shall browse through the images that are available in the system and choose one of

them, uploading it to that particular geo-location.



Appendix D: Database Design Schema

DATA DICTIONARY:

The table descriptions are as follows

1. Authorized user: This table holds the authentication details like Mobile number and

password for every user who logins to the system. This Mobile number and password is



used by the client for login purposes on the Android phone, whenever the user needs to

download images.

Column Name DataType Default Value

MobileNumber bigint[10] Not Null

Password varchar[45] Not Null

Primary Key: {MobileNumber}

Functional Dependency: MobileNumberPassword

Highest Normal Form: BCNF

2. Profile: Each Authorized user has one account. The profile information contains Mobile

number, First name, Last name, E-mail, Mobile_model.



FirstName varchar[45] Not Null

LastName varchar[45] Not Null

E-mail varchar[60] Not Null

Mobile_model varchar[45] Not Null

Primary Key: {MobileNumber}

Functional Dependency: MobileNumber{FirstName, LastName, Mobile_model, E-mail}




3. ImagePlot: An authorized user can upload any number of images. In order to know which

user has uploaded the image at particular Geo Location, we consider Mobile number,

Latitude and Longitude as a primary key.


ImageID integer[5] Not Null

ImageName varchar[60] NotNull

ImageDescription Varchar[300] NotNull

ImageLink Varchar[200] NotNull

Latitude double(20,15) Not Null

Longitude double(20,15) Not Null


CategoryID integer[15] Not Null

Primary Key: {ImageID}

Foreign Key: {CategoryID, MobileNumber}

Functional Dependency: ImageID ImageName, ImageLink, ImageDescription, Latitude,

Longitude, MobileNumber, CategoryID


4. Category: Each image which is placed at a particular geo-location by an Authorized user

has a related Category Name.


CategoryID interger[15] Not Null

CategoryName varchar[45] Not Null



Primary Key: {CategoryID}

Functional Dependency: CategoryID CategoryName


Appendix E: Use case and Sequence Diagrams

Use case Diagram:

Client side:



Kite level written Use Case for displaying images in a virtual world using Augmented Reality technology:

Use-Case Title: Displaying images in a virtual world using Augmented Reality technology

Primary Actor: Mobile User

Level: Kite (Summary)

Stakeholders: Mobile user

Precondition: Mobile users should have a camera phone with android operating system

Minimal guarantee: Mobile users retrieve the geo codes facing camera lens

Success guarantee: Display images at the specific geo code on camera

Trigger: Mobile users access the application utilizing Augmented Reality technology

Main success scenario:1. Mobile Users opens the Augmented Reality application on Android phones using camera

lens to retrieve the geo code locations.2. Mobile Users requests server for the images by sending the geo codes.3. Images are displayed at corresponding geo code locations on camera after receiving it

from the server database.4. Images can be maximized by touching it.5. Images can be downloaded and stored in to the mobile phone.

Error conditions:1. a. Sending geo codes to the server is interrupted

1.a.1 Transaction Rolled back, Mobile User starts again. 1.a.2 Transaction Rolled back, Mobile User quits.

2. a. Receiving images from server is interrupted 2.a.1 Transaction Rolled back, Mobile User starts again. 2.a.2 Transaction Rolled back, Mobile User quits.

3. a. If images are not downloaded 3.a.1 Transaction Rolled back, Mobile User starts again. 3.a.2 Transaction Rolled back, Mobile User quits.



Server side:



Kite level written Use Case for adding and updating images on server side GUI:

Use-Case Title: Adding and updating Images on Server side GUI

Primary Actor: Authorized user

Level: Kite (Summary)

Stakeholders: Authorized user

Precondition: Server side GUI should be embedded with Google maps API.

Minimal guarantee: Accessing the Google maps from server side GUI with internet

Success guarantee: Adding images at the specific geo code location on server side

Trigger: Administrator access the GUI enabled with Google maps API

Main success scenario:1. Administrator logs in to access the server side GUI.2. Pick a geo location to add an image.3. Add images at specific geo code locations using Google maps API.4. Update all the images with the recent one.5. Maintain the server side GUI regularly.

Error conditions:1. a. “Log-in” Use case failure

1.a.1. Authorized user is told the account information is invalid, Authorized user provides the valid information and login successfully.

2. a. Adding images at Geo codes are interrupted 2.a.1 Transaction Rolled back, Administrator starts again. 2.a.2 Transaction Rolled back, Administrator logs out.

3. a. Updating images are interrupted. 3.a.1 Transaction Rolled back, Administrator starts again. 3.a.2 Transaction Rolled back, Administrator logs out.



Sequence Diagram:

Client side:

Sequence diagram for selecting the category:



Sequence diagram for setting radius and perform camera options:



Sequence diagram for authentication:



Sequence diagram for opening downloaded images:

Server side:



Sequence Diagram for successful login onto web application:

Sequence diagram for picking a geo location to upload an image:



Sequence diagram for adding image and choose category:



Appendix F: Screenshots of Client Side and Server Side

Server Side:



Fig 1: Home Page

Fig 2: Login Page



Fig 3: Registration page

Fig 4: After successful login the user can place a marker on the Google Maps and upload an image.



Client Side:

Images of application running on Dell Streak

Fig 5: Initial Screen

Fig 6: Screen when clicked on menu



Fig 7: Screen with list of categories

Fig 8: Screen when clicked on Camera



Fig 9: Screen when clicked on Map

Fig 10: Screen when tapped on the marker. Login page



Fig 11: Screen when clicked download after successful login

Fig 12: Screen when clicked on info menu button



Fig 12: Screen when clicked pattern recognition in camera button

Fig 13: Screen when clicked on Geolocation on Camera menu button


SMAART - dcm.uhcl.edudcm.uhcl.edu/caps11g5/tech.docx · Web viewACKNOWLEDGEMENT. We take an...

Documents

Transcript of SMAART - dcm.uhcl.edudcm.uhcl.edu/caps11g5/tech.docx · Web viewACKNOWLEDGEMENT. We take an...