Alex Karpus Honours Project 2009

AN OPENCV IMPLEMENTATION OF A QR

BARCODE READER FOR THE IPHONE AND

WINDOWS MOBILE PLATFORMS

HONOURS PROJECT

Written By: Alex Karpus

Supervised By: Dr. Gerhard Roth

Carleton University

December 13, 2009

2 ©Alex Karpus, 2009

ABSTRACT

With the ever changing technology market, mobile devices are the latest in powerful, compact

and capable line of consumer electronics. Many existing applications are finding new uses on these

devices. While there are many different areas currently benefiting from the mobility and computational

power offered by today’s mobile devices, the field of Computer Vision in particular, is seeing resurgence

in new innovation targeted for these devices.

This paper details the process of the design, development and implementation of a QR Barcode reader.

The main objective of this paper is to serve as a guide for developing or porting new and existing

computer vision application to the iPhone and Windows Mobile development platforms. As part of this

paper, OpenCV, a leading open source computer vision framework has been ported to work on both the

iPhone and Windows Mobile powered mobile devices.


ACKNOWLEDGEMENTS

I would like to thank Dr. Gerhard Roth for supervising this project, for his continued input along the

way and help with this paper. I hope that the results of this project will be useful to him and his future

students. I would also like to thank Daniele Pizzoni and Yoshimasa Niwa for their great work in porting

OpenCV to the iPhone. Also thank you to Nishi Takao for his work on the libdecodeqr library. As well as

Rhonda Software for providing useful and insightful observations and code for making OpenCV usable

on the iPhone platform.


TABLE OF CONTENTS

Introduction ............................................................................................................................................ 7

Background ............................................................................................................................................ 9

Computer Vision ................................................................................................................................. 9

Barcodes and the QR Code................................................................................................................ 10

OpenCV ............................................................................................................................................ 11

Libdecodeqr ...................................................................................................................................... 12

iPhone SDK ....................................................................................................................................... 13

Windows Mobile SDK ........................................................................................................................ 14

Goals and Objectives ............................................................................................................................ 15

Motivation ........................................................................................................................................ 15

Objectives ......................................................................................................................................... 15

Learning Goals .................................................................................................................................. 16

QR Code Reader Design .........................................................................................................................17

QR Code Structure .............................................................................................................................17

Reader Design ................................................................................................................................... 18

Status Guide .................................................................................................................................. 20

Experiments ...................................................................................................................................... 22

OpenCV on Mobile Devices ................................................................................................................... 24

iPhone ............................................................................................................................................... 24

Prerequisites ................................................................................................................................. 25

Compiling ...................................................................................................................................... 25

Using OpenCV ............................................................................................................................... 26

Windows Mobile ............................................................................................................................... 27

Prerequisites ................................................................................................................................. 28

Compiling ...................................................................................................................................... 29

Using OpenCV ............................................................................................................................... 39

QR Code Reader for the iPhone ............................................................................................................ 41

Design ............................................................................................................................................... 42

User Interface.................................................................................................................................... 44

QR Code Reader for Windows Mobile ................................................................................................... 46


Design ............................................................................................................................................... 47

User Interface.................................................................................................................................... 51

Results .................................................................................................................................................. 53

QR Code Reader Accuracy ................................................................................................................ 53

QR Code Reader Performance .......................................................................................................... 54

Conclusion ............................................................................................................................................ 56

Contributions to the Field.................................................................................................................. 56

Future Work ...................................................................................................................................... 57

References ............................................................................................................................................ 58

Appendices ........................................................................................................................................... 60

Appendix A – CD Contents ................................................................................................................ 60

Appendix B – Test Cases ................................................................................................................... 62

QR Code Test Sample #1 .............................................................................................................. 63




LIST OF FIGURES

Figure 1 - Basic OpenCV structure (Bradski & Kaehler, 2008) ................................................................ 12

Figure 2 - QR Code structure (Wheeler, 2008) ........................................................................................17

Figure 3 - Structure of QR Code reader application ............................................................................... 18

Figure 4 - Windows version of iQR ........................................................................................................ 20

Figure 5 - QR Code reader adaptive threshold ....................................................................................... 22

Figure 6 - CMake configuration window ................................................................................................ 30

Figure 7 - Unloading OpenCV projects in Visual Studio ......................................................................... 32

Figure 8 - Adding a new solution platform in Visual Studio ................................................................... 32

Figure 9 - Visual Studio project Properties dialog .................................................................................. 33

Figure 10 - QR Code reader for the iPhone ............................................................................................ 41

Figure 11 - iQR design for the iPhone .................................................................................................... 42

Figure 12 - UIImage conversion to and from IplImage(Rhonda Ltd, 2009) ............................................. 43

Figure 13 - iQR main screen for the iPhone ........................................................................................... 44

Figure 14 - iQR taking a picture for the iPhone ...................................................................................... 44

Figure 15 - iQR processing image for Windows Mobile .......................................................................... 45

Figure 16- iQR decoded message for the iPhone ................................................................................... 45

Figure 17 - QR Code reader for Windows Mobile ................................................................................... 46

Figure 18 - iQR design for Windows Mobile ........................................................................................... 48

Figure 19 - Marshalling strings to and from unmanaged code ............................................................... 50

Figure 20 - Memory management between managed and unmanaged code ........................................ 51

Figure 21 - iQR main screen for Windows Mobile .................................................................................. 52

Figure 22 - iQR taking a picture for Windows Mobile............................................................................. 52

Figure 23 - iQR processing image for Windows Mobile .......................................................................... 52

Figure 24 - iQR decoded message for Windows Mobile ......................................................................... 52

Figure 25 - QR Code reader performance graph .................................................................................... 54

LIST OF TABLES

Table 1 - Libdecodeqr successful statuses ............................................................................................. 21

Table 2 - Libdecodeqr failed statuses .................................................................................................... 21


INTRODUCTION

Mobile devices have reached a ubiquitous status in society. The number of cell phones, personal

digital assistants (PDAs) and general handheld mobile devices has long surpassed the number of

computers in the world (CIA - The World Factbook - World - Communications, 2009). Most mobile

devices today are empowered by full color displays, integrated digital cameras, fast processors and

even dedicated 3D graphic chips. This technology has allowed these mobile devices to become capable

and sophisticated computing peripherals capable of fulfilling a variety of personal and commercial

purposes.

One purpose that mobile devices have started to rapidly fulfill is in the area of computer vision. For

instance, bar codes can now be read on most mobile devices. Bar codes are simply two-dimensional

machine-readable representations of information, like text, a phone number or a uniform resource

locator (URL). Thus an opportunity exists to take advantage of camera equipped mobile devices for

computer vision development in areas such as barcode reading without the use of lasers as is common

in the commercial sector. A particular type of barcode that is of interest is the Quick Response (QR)

Code. This is a matrix code which is not only efficient for storing large amount of information, but has

also been designed for high-speed decoding.

This paper and the associated deliverables are about the development of a QR code reader for the

Apple iPhone and Microsoft Windows Mobile Smartphone. As part of the project, a foundation of

knowledge was built to allow rapid computer vision development on the both the iPhone and Window

Mobile device platforms using the Open Computer Vision Library (OpenCV). The information in this

paper will help minimize the overhead encountered in porting existing code to the new mobile device

platforms and reduce the development time of new computer vision algorithms for mobile devices.


The accompanying QR code reader serves as an example application, highlighting the possibilities of

OpenCV on the mobile platform.

The next section of this report introduces background information on the technologies and concepts

involved in this project.


BACKGROUND

COMPUTER VISION

Computer vision is a branch of computer science concerned with analyzing images to extract

information about an environment. It may seem that concepts of computer vision are quite easy. After

all, how hard can it be to distinguish between a tree, a car and a barcode all present in the same image?

However, our initial intuitions can be quite misleading because the task of recognizing objects in our

environment is something we take for granted. In fact, something as simple as recognizing an object is

performed by complex and as of yet, little understood system and processes in the brain (Chai, 2009).

Much like the human visual system, computer vision systems can perform the same visual functions

except that what we perceive as colors or patterns, computers see as grids of numbers, which can

represent different colors, luminosities, levels of saturation, and other conceptualized values. The

simple cameras used in computer vision are primitive with no pattern recognition, no automatic

focusing and most importantly, no previous understanding and experience to rely on. Through artificial

intelligence techniques and image processing algorithms, computer vision applications can extract

information about an image.

There are many algorithms that exist today, capable of performing simple tasks like revealing contours

in an image, or balancing the light exposure evenly across an image. The overall task is still quite

daunting. To go from a grid of numbers, into a conceptualized idea about an image is still a very difficult

thing.

With the advent of mobile technology, computer vision now has yet another outlet for testing and

refining its algorithms and ideas.


BARCODES AND THE QR CODE

Originally invented and patented in 1957 by Woodland and Silver, a barcode was envisioned for

automatically reading product information during a grocery store check out (Bellis, 2009). However

since then many different kinds of barcodes have been developed and put to use in a variety of

environments.

In 1994 a Japanese company Denso Wave developed a new type of 2D matrix barcode called the Quick

Response (QR) code. This new type of code has a high capacity for encoding data, prints smaller than

regular barcodes while encoding the same data and due to its built-in error checking design, is quite

damage tolerant. It was initially designed for use in auto part manufacturing, allowing for quick part

identification in an assembly line process. However due to its advantages over typical barcodes its

usage has since expanded into the consumer world and has become particularly wide spread in Japan.

QR Codes now appear on books, magazines, CDs, and other merchandise available to regular

consumers. With the advancements in recent years of the technology behind mobile devices and their

integrated digital cameras, it is now possible to develop consumer-grade mobile device applications

which, using the built in camera technology, can decode barcodes on the fly. Since QR Codes can store

anything from uniform resource locators (URLs), phone numbers and text, their usage has become

widespread.

Although the QR Code has been considerably more popular in Japan than in North American its

usefulness is not lost on major technologically centered businesses. Recently Google, a large American

corporation offering many web services, released their initiative for QR Codes in the United States.

Their plan is to provide the 100,000 most popular local businesses with decals featuring a QR Code. The

businesses are encouraged to place these decals in their windows so that passing customers can use

their mobile phone to find out more information about the local business. The customer can read


reviews, see pictures of the venue or even see any available coupons, all from their mobile device

(Hayward & Kim, 2009).

Because of its wide popularity, and the backing of companies like Google, the QR Code was chosen as

the subject for the application development portion of this paper.

OPENCV

Open Computer Vision Library or OpenCV for short, is an open source computer vision library.

It is a very strong framework which aims to provide a simple-to-use computer vision infrastructure that

allows developers to build sophisticated computer vision applications with little overhead. The library is

written in C with a recently rewritten C++ interface. It runs natively under Windows, Linux and Mac OS

X (Bradski & Kaehler, 2008).

The OpenCV library contains over 500 functions that cover many areas in computer vision, including

camera calibration, stereo vision, robotics, user interface, and factory product inspection. By popular

demand of machine learning algorithms with computer vision implementations, OpenCV also has a

general-purpose machine learning library.


CVImage processing and Vision

Algorithms

MLStatistical Classifiers and

Clustering Tools

HighGUIGUI, Image and Video I/O

CXCoreBasic structures and algorithms, XML support, drawing functions

Figure 1 - Basic OpenCV structure (Bradski & Kaehler, 2008)

The basic structure of OpenCV (Figure 1) is split into four main components; CV, ML, HighGUI and

CXCore. Each component contains a certain set of functionality. The CV component contains

functionality for basic image processing and manipulation as well as computer vision algorithms; ML

contains all the machine learning functionality which includes clustering and statistical classifiers.

HighGUI contains functionality for loading and saving images, video. Finally CXCore is a common

component between all the other components because it contains all the core data structures and

content commonly used by other components.

Due to a plethora of existing work already done with the OpenCV framework, it was chosen for its

robustness and ease of use. The native C/C++ interfaces also made it an ideal choice as both the iPhone

SDK and the Windows Mobile SDK support similar languages.

LIBDECODEQR

Libdecodeqr is a C/C++ library developed by Nishi Takao for the decoding of QR Codes based

on the JIS X 0510 and ISO/IEC18004 standards. The library relies on OpenCV for the majority of its


image processing functionality. It can be easily integrated into various platforms for high speed

decoding (Takao, 2007).

While it was feasible to implement a QR Code decoder from scratch, a document describing the QR

Code implementation would be required. Due to the cost of the document describing the encoding and

time required to implement already existing functionality, libdecodeqr was chosen to aid in reducing

the development time required for implementing a useful mobile QR Code reader. Its reliance on

OpenCV made it an ideal choice.

IPHONE SDK

The Apple iPhone SDK is a complete software developer kit (SDK) used in the creation of

software for the iPhone OS. The SDK has four distinct layers, which are the Core OS, Core Services,

Media, and Cocoa Touch (Apple Inc, 2009).

The Core OS layer is composed of the XNU kernel, drivers, and basic interfaces of the iPhone OS. All

virtual memory system, file system, threading, inter-process communications, and network are

managed in this Core OS layer. It also provides functionality for interacting with the iPhone’s hardware

components.

The Core Services layer provides core system services and data structures that are shared by all

applications. Basic data management, including data types and collections, bundles, string

management, and more are available to the developer through this layer.

The Media layer houses all of its graphic, audio, and video functionality. This layer allows developers to

extend the standard user interface from 2D to 3D rendering with support for animations, audio tracks,

and video playback.


The Cocoa Touch layer provides developers with access to user interface frameworks like with support

for things like application management, window support, text and web content support, and gestures

with motion based events.

The SDK supports C, C++ and Objective-C languages and for the purposes of this project, only

Objective-C was used.

WINDOWS MOBILE SDK

The Microsoft Windows Mobile SDK is a complete software developer kit used in the creation of

software for the Windows Mobile OS. The latest version of the developer kit and the one used in this

project is version 6. The SDK is divided into three different types of deployments, Professional,

Standard, and Classic. The difference in the types lies in the hardware that an application was

developed for. Professional referring to mobile devices that include cellular phone functionality, while

classic is used solely by PDAs.

All of the work done in this project was developed and tested on a Palm Treo 700wx Smartphone

running Windows Mobile 6 which falls under the Professional deployment category type. The SDK is

further divided into many distinct components of functionality like CellCore for wireless connection

oriented services, ActiveSync for synchronization of data between your computer and mobile device

and many others (Microsoft, 2008).

The SDK supports application development using both C++ and C# languages. The Windows Mobile

platform supports the .NET Compact Framework. This project relies on both of these languages.


GOALS AND OBJECTIVES

MOTIVATION

The rapid growth of the mobile device market in the last couple of years has created new

opportunity for innovation. In particular the area of Computer Vision has been experiencing resurgence

in interest as established algorithms and applications are finding new life on mobile devices.

This project was inspired by the emerging need for simplifying the migration path of existing

applications onto mobile devices. Since the mobile device market has multiple competing devices with

very different development platforms, a roadmap for simplifying the transition was something that I

felt would be useful both for Carleton University’s future mobile development stream, as well as the

open source community involved in OpenCV development.

It is my hope that my work will reduce the development time for individuals interested in computer

vision related projects on the popular iPhone and Windows Mobile platforms.

OBJECTIVES

The motivation for this project has some associated objectives that were completed and are

outlined in this paper.

1. Research existing QR Code systems. Implement a QR Code reader with basic context

awareness using the computer vision OpenCV framework.

2. Research and develop a roadmap for the work required in porting existing OpenCV applications

to mobile devices. In particular focusing on the iPhone and Windows Mobile device platforms.


3. Successfully port the QR Code reader application developed as per the first objective to the

iPhone and Windows Mobile device platforms in order to demonstrate the usability and

completeness of the second objective.

LEARNING GOALS

My personal motivation in taking on this project was to learn and accomplish several things.

Mainly I have been fascinated by the growth of the mobile device market and thus wanted to learn the

skills necessary for developing mobile applications on the popular mobile platforms. Prior to this

project I had rudimentary mobile development experience from various personal projects which I

developed using Java2ME. However the mobile device market has changed a lot and so have the

dominant platforms. As such I wanted to learn about the iPhone SDK and the Objective-C language,

which as of yet, I had never worked with in either my personal or professional careers. As well the

Windows Mobile platform and the C++/C# languages were already familiar to me but I had never

actually gone through the phases of development on this platform.

I was also interested in the difference between mobile (embedded) architecture and the conventional

architecture of everyday computers when it came to reusing existing code on new devices. This is why I

felt that a great way to learn about differences and similarities would be to go through the work

required in successfully porting existing code to a new platform.


QR CODE READER DESIGN

QR CODE STRUCTURE

Figure 2 - QR Code structure (Wheeler, 2008)

In order to understand the QR Code reader (iQR) design it helps to understand a little about the

structure of the QR Code.

The QR Code region is made up for three position tags located in the upper left, bottom left and upper

right corners. This is in fact a unique feature which can be used to distinguish QR Codes from other 2D

barcodes. These are used for locating the QR Code in an image and help serve as markers. They are also

helpful in determining the degree of skew and rotation of the code in the image. There is also an

alignment tag, which can be used for further alignment calculations.

For redundancy, the version and format information is stored in at least two places around the position

tags. This allows for better data recovery should a corner of the code be damaged or obstructed.

Because different types of data can have different representations in the QR Code, it is vital that as part

of the decoding process, the format information be recoverable. Otherwise the encoded data is left

unreadable. The version information can be used for backward capability, in supporting older revisions

of the encoding protocol.


The checkered pattern present, both vertically and horizontally between two neighboring positions

tags allows for a calibration routine to perform calculations to determine the size of single black and

white regions.

One of the main strengths of the QR Code is its multiple levels of error correction. It uses a Reed-

Solomon error correction technique and depending on the level of data corruption can support up to

30% of data restoration. Due to the complexities involved full error correction support is not built into

libdecodeqr.

READER DESIGN

iQR

libdecodeqr

Decode Image

OpenCV

Acquire Image

Display Results

Process Image

Figure 3 - Structure of QR Code reader application

The QR Code reader (iQR) is an application which utilizes both libdecodeqr and OpenCV in

order to decode an image containing a QR Code.


The design was intended for portability between platforms. Once ported to platform, a user interface

can be wrapped around the basic design to simplify the ease of use and better deliver decoded results.

A more detailed process flow shown in Figure 3 is described below.

1. Acquire image. – Depending on the platform, the image can come either from file, camera or

image library of the device. The actual implementation of acquiring an image is platform

dependent.

2. Handover the acquired image to the Libdecodeqr library for image processing. – All of

libdecodeqr is implemented using OpenCV functions. Thus due to OpenCV’s portability (See

the OpenCV on Mobile Devices section on page 24), this part is platform independent.

a. Apply adaptive binary thresholding. – Image processing technique which reveals image

detail hindered by poor or uneven lighting conditions.

b. Fix the effects of skew and camera rotation on the QR Code. – The three position tags

and alignment tag are used to extract the QR Code and fix perspective issues (skew,

rotation).

c. Decode the resulting image.

3. Display results to the user.


Figure 4 - Windows version of iQR

The Windows version (Figure 4) works by reading in an image from a file and processing it. In the

example figure, a generated QR Code containing a URL was fed into the program. The program was

able to locate the position tags, find and extract the QR Code from the image and print out the decoded

results.

See the QR Code Reader for the iPhone section on page 41 and the QR Code Reader for Windows

Mobile section on page 46 for the mobile versions of this application.

STATUS GUIDE

Once the QR Code is analyzed and processed, the decoded data (if any) is accompanied by a

status code. Depending on the success of the decoding process, the status helps debug where in the

process decoding difficulties were encountered.


Following are two tables highlighting the different possible statuses.

Successful Status (base 16) Description

2000 Image decoded

0f00 Genera version info error.

0100 Version info mismatch.

0800 Unrecoverable version info.

00f0 General format info error.

0010 Invalid format level used.

0080 Unrecoverable format info.

000f General data error.

0001 Not supported data format.

0002 Length of data stored not as expected.

0008 Unrecoverable data.

Table 1 - Libdecodeqr successful statuses

Failed Status (base 16) Description

4000 Image could not be decoded.

0100 Unsupported image format.

0200 Position tags could not be found.

0400 Code area could not be found.

0800 Code area could not be determined.

Table 2 - Libdecodeqr failed statuses

The first table (Table 1) is a collection of statuses returned when a QR Code was found in the image and

an attempt at decoding it was made.

The second table (Table 2) is a collection of statues returned when a QR Code cannot be located in the

image being decoded. In each case, the first row of the table contains the main status to which the

entries following it are logically added.

For example, a status of 0x2089 would indicate that the image was processed, QR Code was found,

however the format info could to be recovered and neither could the data (perhaps because the data

format was unsupported).


EXPERIMENTS

As part of testing the QR Code reader design and reliability, several attempts were made at

improving the accuracy and performance aspects of the design. One such experiment was in testing the

affects of applying adaptive light thresholding and perspective skew adjustment based on the location

and positioning of the three corner position tags of the source image.

Using OpenCV’s implementation of the adaptive thresholding algorithm the source image was first

adjusted and then it was allowed for the libdecodeqr code to try and detect the image.

Figure 5 - QR Code reader adaptive threshold

This two step process would allow for the position tags to be found. Using the position tags, it would be

possible to estimate the rotation of the QR Code in the image. Measuring the displacement of the top

two position tags and the two left position tags would allow for the calculation of the angle or rotation.

Also the stretch of the image could be approximated by measuring the distance between the top two

position tags and comparing that to the distance between the two left position tags. Scaling the

resulting image to the larger of the two would help eliminate simple stretch.


However, having applied these basic techniques, to the image prior to the decoding process led to the

discovery of little to no improvement. The result of this was due to the fact that the libdecodeqr already

implemented a similar approach for handling lighting and rotational/skew effects on the source image.

Figure 5 shows a screenshot of what an original input image looks like compared to an image where

adaptive thresholding and minor rotation compensation has been applied.


OPENCV ON MOBILE DEVICES

When this project was first envisioned, information about OpenCV on mobile devices existed

but was quite scarce. This was one of the motivating factors for bringing about a solution that would

contribute to the field and offer a chance to learn about the emerging mobile market.

There are several things to consider when porting existing code to new environments. In order to

successfully cross compile code for a new system, it is important to know what architecture is used by

the system. The system architecture commonly used in mobile devices currently on the market is the

Advanced RISC Machine (ARM) architecture. It was developed by ARM Limited and originally intended

for desktop computers. However the success of the x86 IBM PC has pushed ARM’s efforts into the

mobile and embedded device markets instead (ARM architecture, 2009).

Both the iPhone and the Windows Mobile development platforms offer full ARM processor support and

thus share similar paths for porting OpenCV.

IPHONE

Up to now I have referred to the iPhone as a single device. When in fact there have been three

different versions of the iPhone as of the time of this paper. All of the development involved in this

project was carried out on the iPhone 3G. From now on, all information provided will be specific to the

iPhone 3G.

The iPhone use a 412MH ARMv6KZ processor from the ARM11 family of architecture. It has 128MB of

RAM and 8GB of internal storage. The primary digital camera has a 2MP CMOS sensor capable of

producing images of up to 1600x1200 pixels in size, however there no video support (iPhone, 2009).


The sections that follow outline the prerequisites and give instructions for compiling and using OpenCV

2.0 in iPhone applications.

PREREQUISITES

To successfully compile OpenCV for the iPhone you will need:

- An Intel powered Macintosh computer running Mac OS X 10.5+ (Leopard tested)

- The iPhone SDK 3.0+ developer environment and tools. (iPhone SDK 3.0 tested)

» http://developer.apple.com/iphone/

- OpenCV 2.0.0 (OpenCV-2.0.0.tar.bz2, 2009-10-01 tested)

» http://sourceforge.net/projects/opencvlibrary/files/

COMPILING

In order to compile OpenCV into static universal libraries, follow the steps outlined here.

1. Create a staging directory.

2. Extract OpenCV into the staging directory.

3. Place the `build_opencv.sh` iPhone build script into the staging directory (See Appendix A on

page 60 for information on the scripts accompanying this document).

4. Open a Terminal window.

5. Navigate to the staging directory.

6. Run the `build_opencv.sh` script. This script will attempt to patch and build OpenCV. The

whole build process can take up to 15 minutes.

7. Once everything is completed, the staging directory will contain three new folders.

a. `opencv_simulator` - Contains the static OpenCV libraries which can only be used for

testing in the iPhone simulator.


b. òpencv_device` - Contains the static OpenCV libraries which can only be used on

iPhone hardware.

c. òpencv_universal` - Contains the static OpenCV libraries which are a hybrid of the

libraries found in the previous two folders. These libraries can be used both in the

simulator and on the hardware.

The structure of the òpencv_universal` folder is as follows:

- opencv_universal

- libs – Contains the universal static libraries.

- libcv.a

- libcxcore.a

- libcvaux.a

- libml.a

- libhighgui.a

- include – Contains all the relevant OpenCV header files.

USING OPENCV

The steps outlined in the previous section will have produced five static libraries, which can now

be integrated into your development environment.

The Mac OS X Developer toolkit comes with an IDE called Xcode. Since Xcode is the most widely used

IDE, the steps outlined below are tailored to it. However the libraries generated in the previous section

can be used with any build environment.

The instructions that follow assume that an Xcode project already exists. If not create it now.

1. Copy the òpencv_universal` folder generated in the previous section, into your Xcode

project directory.

2. Launch Xcode and open the project.

3. From the menu, go to Project > Edit Project Settings

4. Select the Build tab


5. Add the following flags to the `Other Linker Flags` Linking setting

“$SRCROOT/opencv_universal/libs/libcv.a”

“$SRCROOT/opencv_universal/libs/libcxcore.a”

“$SRCROOT/opencv_universal/libs/libcvaux.a”

“$SRCROOT/opencv_universal/libs/libml.a”

“$SRCROOT/opencv_universal/libs/libhighgui.a”

–lz

–lstdc++

6. Add the following path to the `Header Search Paths` Search Paths setting:

“$SRCROOT/opencv_universal/include/opencv”

You are now ready to use OpenCV in your project. Don’t forget to include the OpenCV header files in

your code:

#include “cv.h”

#include “cxcore.h”

#include “highgui.h”

See the QR Code Reader for the iPhone section on page 41 for a demonstration of the implementation

and usage of OpenCV on the iPhone.

WINDOWS MOBILE

Unlike the Apple iPhone and the iPhone SDK, Microsoft’s Windows Mobile operating system

and associated environment give developers one environment in which to make mobile applications

that will work on a multitude of hardware devices.

This is because of a fundamentally different approach that Microsoft takes to software development, as

opposed to Apple. Microsoft strives to make its software work on a vast array of devices which can

support the Windows Mobile OS. Apple on the other hand, controls the hardware and the software

giving little choice to the end consumer. Although there are downsides to Microsoft’s approach, just as

there are downside’s to Apple’s. However, I believe that the versatility of Windows Mobile is a greater

asset because it allows for a single developer environment to create applications for differently


configured mobile devices. This is very important, especially for educational institutions, such as

Carleton University, or any college or university interested in offering a mobile development program.

All the Windows Mobile work for this project was done on a Palm Treo 700wx running Windows Mobile

6. Similarly to the iPhone, the Treo uses a 312MHz PXA270 Intel XScale processor based on the

ARMv5TE family of processors. The phone has 128MB of memory, of which 62MB can be used for

program and data storage. It has a 1.3 MP digital camera capable of producing images of 1280x1024

pixels in resolution with up to 2x digital zoom (Kairer, 2006).

The sections that follow outline the prerequisites and give instructions for compiling and using OpenCV

2.0 in Windows Mobile applications. Unlike for the iPhone not all of OpenCV has been completely

ported to Windows Mobile. Only the main components CV, ML, HighGUI and CXCore are ported as per

the following sections. Also important to note that only image loading/saving capability of HighGUI was

ported over, excluding OpenCV video and camera support.

PREREQUISITES

To follow the instructions outlined here you will need:

- A computer running Microsoft Windows XP or newer (Windows XP tested)

- Microsoft Visual Studio 2008 or newer (Visual Studio 2008 tested)

- Cross Platform Make (CMake) 2.6+ (CMake 2.8 tested)

» http://www.cmake.org/

- Patch GnuWin32 (version 2.5.9 tested)

» http://gnuwin32.sourceforge.net/packages/patch.htm

- Windows CE C Library Extensions (wcelibcex-1.0.zip)

» http://wcelibcex.sourceforge.net/

Menna

Highlight

Menna

Highlight

Menna

Highlight

Menna

Highlight

Menna

Highlight


- The Windows Mobile 6.0+ SDK developer environment and tools. (Windows Mobile SDK 6

tested)

» http://www.microsoft.com/downloads/details.aspx?familyid=06111A3A-A651-4745-88EF-

3D48091A390B&displaylang=en

- OpenCV 2.0.0 (OpenCV-2.0.0.tar.bz2, 2009-10-01 tested)

» http://sourceforge.net/projects/opencvlibrary/files/

COMPILING

1. Create a staging directory.

2. Extract OpenCV into the staging directory.

3. Place the `opencv_winmo.patch` patch file into the staging directory (See Appendix A on page

60 for information on the files accompanying this document).

4. Open a Command Prompt window.

5. Navigate to the staging directory and then into the OpenCV directory.

6. Run the following command. It will patch the OpenCV source and make it Windows Mobile

friendly. It should be noted that as part of this patch a lot of highgui functionality is removed

due to incompatibility.

patch –p1 < ../opencv_winmo.patch

7. Start CMake (cmake-gui)

Menna

Highlight

Menna

Highlight

Menna

Highlight

Menna

Highlight

Menna

Highlight

Menna

Highlight

Menna

Highlight

Menna

Highlight

Menna

Highlight

Menna

Highlight

Menna

Highlight


Figure 6 - CMake configuration window

8. Click `Browse Source` to browse for the OpenCV source code. It should be within the staging

directory.

9. Click `Browse Build` and set the folder where you want the configuration and compiling to

take place. I recommend a folder on the same level as the OpenCV code inside the staging

directory.

10. Click `Configure`.

11. The CMake system will prompt to select a generator for the project. Choose the build system

you are going to use. This guide assumes you will use Visual Studio 9 (2008).

12. In the same prompt as in the previous step, select the `Use default native compilers`

option.

13. Click `Finish` to proceed.

14. It may take a moment for your build system to be verified. Once the process is complete the

CMake window will look similar to Figure 6.

15. Set the following environment variables in the CMake window.

CMAKE_CXX_STANDARD_LIBRARIES =coredll.lib corelibc.lib ole32.lib oleaut32.lib uuid.lib commctrl.lib

Menna

Highlight

Menna

Highlight

Menna

Highlight

Menna

Highlight

Menna

Highlight

Menna

Highlight

Menna

Highlight

Menna

Highlight


CMAKE_C_STANDARD_LIBRARIES =coredll.lib corelibc.lib ole32.lib oleaut32.lib uuid.lib commctrl.lib

CMAKE_EXE_LINKER_FLAGS = /STACK:10000000 /machine:thumb

CMAKE_MODULE_LINKER_FLAGS = /STACK:10000000 /machine:thumb

CMAKE_SHARED_LINKER_FLAGS = /STACK:10000000 /machine:thumb

ENABLE_OPENMP =0

BUILD_TESTS =0

BUILD_NEW_PYTHON_SUPPORT =0

16. Click `Configure` in the CMake window. This will enable you to generate your Visual Studio

solution.

17. Click `Generate`. The process will take a few moments and after it is done, you can exit

CMake.

18. Navigate to your build directory.

19. Extract the Windows CE C Library Extensions archive into the build directory.

20. Build the Windows CE C Library Extensions library with the Windows Mobile 6 Professional

SDK. If you are unsure of how this is done, skip this step and return to it once you’re ready to

compile libpng. This will you give you a chance to complete more of this guide and give you

some experience with compiling for Windows Mobile.

21. Open the OpenCV solution file. This will launch Visual Studio.

22. Now in Visual Studio, in the Solution Explorer window, unload all the associated OpenCV

projects. Refer to Figure 7.

Menna

Highlight

Menna

Highlight

Menna

Highlight

Menna

Highlight

Menna

Highlight

Menna

Highlight

Menna

Highlight


Figure 7 - Unloading OpenCV projects in Visual Studio

23. Now select and reload only the following projects: cv, cxcore, flann, highgui,

libjpeg, libpng, ml, opencv_lapack, zlib. This minimizes the compile time and

includes only the essentials.

24. Right-Click on the `HighGui` project and set it as the startup project.

25. Open the Configuration Manager from the Build > Configuration Manager menu.

26. Add a new solution platform for the Windows Mobile 6 Professional SDK (ARMV4I). See Figure

8 for details.

27. Set the active solution configuration to be `Release` and close the configuration manager.

Figure 8 - Adding a new solution platform in Visual Studio

Menna

Highlight

Menna

Highlight

Menna

Highlight

Menna

Highlight

Menna

Highlight


What follows now is a breakdown by component of the changes required to compile it. The list number

continues, implying the order should be kept when following the guide.

All the sections below deal with individual project properties. In order to better understand the

instructions, the terminology is briefly described in the next section.

Individual Project Properties

The project properties dialog can be opened by selecting a project from the Solution Explorer

and Right-Clicking on it and selecting `Properties`, or alternatively from the Project >

Properties menu.

The Properties dialog in Visual Studio 2008 looks like Figure 9.

Figure 9 - Visual Studio project Properties dialog

The dialog has several important areas. On the left hand side is a tree-view list of the different project

areas that can be configured. The different areas control everything from the options used for

compiling a project, the way it is debugged, to the output directory.


The majority of the options will be left untouched. However there are a few critical options that must be

set manually.

In the later sections of this guide, when it says “Set the `Preprocessor Definitions` in the C/C++

Preprocessor section.” What it is referring to is the C/C++ section in the left hand side of the dialog.

Underneath which is a Preprocessor child, which when clicked on, reveals a slew of properties, one of

them being the Preprocessor Definitions.

Once you’re done making changes, simply click ‘OK’ and the changes will be saved.

Now continue to the next sections for compiling individual parts of OpenCV.

Compiling Zlib

28. Add the following definitions to the `Preprocessor Definitions` in the C/C++ Preprocessor

section.

NO_ERRNO_H

_WIN32_WCE=$(CEVER)

UNICODE

_UNICODE

29. Build zlib. The build should complete without errors.

Compiling Opencv_lapack


section.

UNDER_CE

_WIN32_WCE=$(CEVER)

UNICODE

_UNICODE

31. Build opencv_lapack. The build should complete with some warnings but no errors.

Menna

Highlight

Menna

Highlight

Menna

Highlight

Menna

Highlight

Menna

Highlight

Menna

Highlight

Menna

Highlight


Compiling Flann


section.

WINCE

_WIN32_WCE=$(CEVER)

$(ARCHFAM)

$(_ARCHFAM_)

UNDER_CE

UNICODE

_UNICODE

33. Build flann. The build should complete with some warnings but no errors.

Compiling Cxcore


section.

WINCE

_WIN32_WCE=$(CEVER)

$(ARCHFAM)

$(_ARCHFAM_)

UNICODE

_UNICODE

UNDER_CE

35. Set the `SubSystem` in the Linker System section to `WindowsCE`.

36. Add `secchk.lib` to `Ignore Specific Library` in the Linker Input section.

37. Build cxcore. The build should complete with some warnings but no errors.

Compiling Cv


section.

WINCE

_WIN32_WCE=$(CEVER)

UNDER_CE

UNICODE

Menna

Highlight

Menna

Highlight

Menna

Highlight

Menna

Highlight

Menna

Highlight

Menna

Highlight

Menna

Highlight

Menna

Highlight

Menna

Highlight

Menna

Highlight

Menna

Highlight


_UNICODE

$(ARCHFAM)

$(_ARCHFAM_)


40. Build cv. The build should complete with some warnings but no errors.

Compiling Libjpeg


section.

NO_GETENV

UNICODE

_UNICODE

WINCE

UNDER_CE

$(ARCHFAM)

$(_ARCHFAM_)

_WIN32_WCE=$(CEVER)

42. Build libjpeg. The build should complete with some warnings but no errors.

Compiling Libpng


section.

_WIN32_WCE=$(CEVER)

$(ARCHFAM)

$(_ARCHFAM_)

UNDER_CE

WINCE

UNICODE

_UNICODE

44. Add the path to the Windows CE C Library Extensions source directory to the `Additional

Include Directories` property in the C/C++ General section.

45. Build libpng. The build should complete with some warnings but no errors.

Menna

Highlight

Menna

Highlight


Compiling Ml


section.

WINCE

_WIN32_WCE=$(CEVER)

$(ARCHFAM)

$(_ARCHFAM_)

UNDER_CE

UNICODE

_UNICODE


48. Build ml. The build should complete with some warnings but no errors.

Compiling Highgui

49. Exclude all source files that start with `cvcap` from the highgui project.


section.

WINCE

_WIN32_WCE=$(CEVER)

UNDER_CE

UNICODE

_UNICODE

$(ARCHFAM)

$(_ARCHFAM_)

51. Remove the following definitions from the `Preprocessor Definitions` in the C/C++ Preprocessor section.

HAVE_TIFF HAVE_JASPER

52. Add the path to the Windows CE C Library Extensions source directory to the `Additional

Include Directories` property in the C/C++ General section.


Menna

Highlight

Menna

Highlight


54. Remove the following libraries from the property Àdditional Dependencies` in the Linker

Input section.

..\..\3rdparty\lib\Release\libtiff.lib

..\..\3rdparty\lib\Release\libjasper.lib

comctl32.lib

gdi32.lib

vfw32.lib

55. Add the path to the Windows CE C Library Extensions library to the property Àdditional

Dependencies` in the Linker Input section.

\wcelibcex-1.0\msvc80\Windows Mobile 6 Professional SDK (ARMV4I)\Release\wcelibcex.lib

56. Build highgui. The build should complete with some warnings but no errors.

Creating Directory Structure

The previous steps were intended to build OpenCV for Windows Mobile 6 ARM architecture. Now in

order to help use these libraries in future projects, a useful directory structure needs to be created.

57. In the staging directory, create a new folder called òpencv_winmo`.

58. Copy the three dll files found in the build directory under `bin/Release` into the

òpencv_winmo\dll` directory.

59. Copy the three lib files found in the build directory under `lib/Release` into the

òpencv_winmo\lib` directory.

60. Copy the include files found in the OpenCV directory under ìnclude/opencv` into the

òpencv_winmo\include`.

You have now created a distributable directory which contains everything required for using OpenCV

on the Windows Mobile 6 platform.

You may have noticed that we did not compile two versions of OpenCV like we did with the iPhone.

This is because the Windows Mobile emulator is an actual hardware emulator. This means that binaries

Menna

Highlight

Menna

Highlight

Menna

Highlight


compiled for devices can be tested in the emulator. This helps eliminate the need for multiple binaries

for different testing environments.

USING OPENCV

The steps outlined in the previous sections will have produced three dynamically linked

libraries, which can now be integrated into your development environment.

As before, the guide to using OpenCV on Windows Mobile devices relies on the Visual Studio

development environment. With Visual Studio, tasks like deployment are vastly simplified. However

the libraries generated in the previous sections can be used with any build environment.

The instructions that follow assume that a Visual Studio project already exists. If not create it now.

1. Copy the òpencv_winmo` folder generated in the previous section, into your Visual Studio

project directory.

2. Launch Visual Studio and open your project.

3. Open the Project Properties dialog. (See the Individual Project Properties section on page 33 for

information about the Properties dialog.)

4. Add the following to the Àdditional Files` property of the Deployment section. This will

ensure that during deployment to the device, the OpenCV libraries are included as part of the

transfer. (As part of the deployment of OpenCV libraries it is important to include the `msvcr90`

library. OpenCV relies on it and your application will not work without it. For more information,

refer to http://msdn.microsoft.com/en-us/library/abx4dbyh.aspx.) The example strings below

are from the iQR project. As such you should replace the string ìQR` with the deployment

name of your project.


highgui200.dll|$(SolutionDir)\opencv_winmo\dll|%CSIDL_PROGRAM_FILES%\iQR|0

cv200.dll|$(SolutionDir)\opencv_winmo\dll|%CSIDL_PROGRAM_FILES%\iQR|0

cxcore200.dll|$(SolutionDir)\opencv_winmo\dll|%CSIDL_PROGRAM_FILES%\iQR|0

ml200.dll|$(SolutionDir)\opencv_winmo\dll|%CSIDL_PROGRAM_FILES%\iQR|0

msvcr90.dll|$(BINDIR)\$(INSTRUCTIONSET)\|%CSIDL_PROGRAM_FILES%\iQR|0

5. Add the following paths to the property `Additional Dependencies` in the Linker Input

section. This will ensure that the linker is able to load the proper definitions during compile

time.

"$(SolutionDir)\opencv_winmo\lib\cv200.lib" "$(SolutionDir)\opencv_winmo\lib\cxcore200.lib" "$(SolutionDir)\opencv_winmo\lib\highgui200.lib" "$(SolutionDir)\opencv_winmo\lib\ml200.lib"

6. Add the path the OpenCV includes to the `Additional Include Directories` property in

the C/C++ General section.

$(SolutionDir)\opencv_winmo\include

You are now ready to use OpenCV in your project. Don’t forget to include the OpenCV header files in

your code:

#include “cv.h” #include “cxcore.h” #include “highgui.h”

See the QR Code Reader for Windows Mobile section on page 46 for an overview using OpenCV on a

Palm Treo 700wx to develop a QR Code reader.


QR CODE READER FOR THE IPHONE

There were no real surprises when developing the iPhone version of the QR Code reader. A lot

of guessing about whether or not OpenCV would work on the actual hardware was dispelled early on in

the project by successful tests. This left a lot of time for developing.

Figure 10 - QR Code reader for the iPhone

Although, iPhone development is quite different from other platform development, in that it forces the

developer to maintain a model-view-controller (MVC) design pattern. While some may argue that this is

a great design pattern to abide by, it definitely has a significant learning curve when you’re unaware of


the pattern in place. Having done a lot Windows development in the past, and now developing on the

iPhone for the first time, there were quite a few things that felt counter intuitive in terms of how to

build even the most basic application.

Part of the large learning curve also comes from the fact that iPhone development requires at least

rudimentary knowledge of Apple’s Objective-C programming language. Objective-C is a reflective and

object-oriented programming language which uses a Smalltalk inspired messaging convention.

DESIGN

As mentioned in the previous section, iPhone development is done mostly in the Objective-C

language. Since Objective-C is a strict superset of the C programming language means that it is

possible to include and successfully compile C code in Objective-C classes (Apple Inc, 2009). This

characteristic of Objective-C meant that porting the QR Code reader code developed in Windows (See

the QR Code Reader Design section on page 17) was simple.

iQR

libdecodeqr

OpenCV

Functions

Figure 11 - iQR design for the iPhone

In comparison to the Windows Mobile version, the iPhone design of the QR Code reader is quite simple

as shown in Figure 11. One thing to notice is that OpenCV is labeled as being part of the iQR binary. This


is because OpenCV for the iPhone can be compiled only as a statically linked library. This means that

during the linking of the iQR application, all the relevant symbols from OpenCV are included in the final

iQR binary. It follows that, unlike the Windows Mobile version, the final binary has no external

dependencies. The downside is that increases the final binary’s file size although this is of no

consequence.

One interesting issue that was isolated to the iPhone implementation had to do with loading images.

The OpenCV function that is often used to loading an image for OpenCV processing is `cvLoadImage`.

Even though this function behaves as expected on both Windows, Windows Mobile and even in the

iPhone simulator; it does not work as expected on the iPhone itself. To work around this issue a set of

functions was written to convert the iPhone SDK’s UIImage class, which represents an image, to the

OpenCV’s IplImage class and vice versa. These functions are stored in the `Function` component of

Figure 11.

- (IplImage *) getCVImageFromCGImage:(CGImageRef)cgImage;

- (CGImageRef) getCGImageFromCVImage:(IplImage *)cvImage;

Figure 12 - UIImage conversion to and from IplImage(Rhonda Ltd, 2009)

One caveat that was discovered from testing was that after processing and extracting the QR Code

from the original image it always came with a black area not filled with any image data. This turned out

to be because libdecodeqr set a region of interest (ROI) on the returned image, but did not actually crop

the image. Thus `getCGImageFromCVImage` had to be modified to abide by ROI restrictions.


USER INTERFACE

The user interface was designed based on the flip-view model commonly used by utility

applications. Upon startup, the application opens the digital camera window allowing the user to take a

photo without having to click unnecessary buttons. After an image is selected it proceeds to attempt to

decode it. If the image is able to be decoded, a new view is displayed with the isolated QR Code, the

status of the decode process, the time it took to decode and the decoded message. If the decoded

message is recognized as a URL, an option appears to navigate to that URL. Below is a collection of the

different views.

Figure 13 - iQR main screen for the iPhone Figure 14 - iQR taking a picture for the iPhone


Figure 15 - iQR processing image for Windows Mobile Figure 16- iQR decoded message for the iPhone


QR CODE READER FOR WINDOWS MOBILE

Although the QR Code reader was designed with portability in mind (see QR Code Reader

Design on page 17) there were some major hurdles that had to be overcome in order to produce a

simple solution.

One of the major hurdles was porting OpenCV to Windows Mobile. Unlike for the iPhone, there was no

information out there on how to compile OpenCV for Windows Mobile 6. In particular at the time of this

project, OpenCV version 2.0 was released, which meant that any and all previous information on the

subject was obsolete by the major changes introduced in the new release.

Figure 17 - QR Code reader for Windows Mobile


DESIGN

The second hurdle that had to be overcome was how to design the actual application. When

developing for the Windows Mobile platform, developers have two choices. Development can be done

in native C++ code, or in managed code leveraging the .NET Compact Framework.

Native code is used to refer to code that when compiled issues commands directly to the CPU using a

CPUs instruction set. When a C/C++ application is compiled, the binary that is produced is made up of

machine level instructions. However managed code is quite different. Managed code is a term used to

refer to code which when run has to be interpreted by yet another runtime. This runtime is called the

interpreter, sometimes also called a just-in-time compiler, will processes the code and produces native

instructions which are then executed just as native code would be. This creates a level of abstraction

which is required for ensuring that the code that is being executed is type and memory safe and

provides the foundation for useful language features like garbage collection. Microsoft’s version of

managed code comes in the form of the .NET Framework which uses the C# programming language.

The managed code generated by compiling C# results in byte code which then is interpreted by the

Common Language Runtime (CLR) layer and converted into CPU instructions (native code) at runtime.

In the case of this project, both OpenCV and libdecodeqr libraries were written in native C/C++.

Conceptually, the simplest thing to do was to develop a user interface in native C++ (ATL) and integrate

calls to OpenCV and libdecodeqr. However after some investigation, it became clear that user interface

development in C++ is quite cumbersome and complicated. Thus the alternative was to turn to the .NET

Compact Framework and design the user interface using the simple drag-and-drop approach. However

this exposed the challenge of interoperability between managed and unmanaged code. After some

research and testing, a simple and elegant solution emerged.


OpenCV

C++ Code

Managed Unmanaged

.NET Compact Framework

iQRApplication (UI)

C# Code

iQRDLL

C++ Code

Libdecodeqr

Figure 18 - iQR design for Windows Mobile

One of the main strengths of the common language runtime (CLR) and the .NET Framework is the

interoperability support between natively compiled code and code converted to native during runtime

using a just-in-time compiler (JIT). Since the .NET Compact Framework is a subset of the .NET

Framework, it has very strong support for managed to unmanaged code interoperability (Keserovic,

Mortenson, & Nathan, 2003).

The solution to the challenge was to use, what is known as, Platform Invoke (P/Invoke). It is a service

provided by the CLR interop layer for calls to unmanaged APIs from managed code. The only

requirement is that the signature of the function being called, be re-declared in managed code.

Applying this to the QR Code reader, as illustrated by Figure 18, the required interaction between

managed code (iQR Application) and unmanaged code (iQR Dll, OpenCV) is all handled by the .NET

Compact Framework.


The managed iQR application is accompanied by an unmanaged dll. This dll exports a single function.

Inside this function all the work relevant to the QR Code process is performed (see Figure 3 on page 18).

This involves calling libdecodeqr (which is linked in during the building of the iQR DLL) and exported

OpenCV functions. Another alternative to creating a dll separate from the main iQR application, was to

make direct calls into OpenCV using P/Invoke, however this would require for the re-declaration of

every OpenCV function that would be used, thus creating a lot of unnecessary overhead. Instead the

solution has a single function re-declaration. Below are code samples of what the declaration of the

main function looks like in the dll and in the managed C# application.

In iQR DLL (DLL.h):

/**

* Contains all the QR Code decoding logic.

*/

extern "C" DLL_API short processImage( const char * in_file,

const char * out_file );

In iQR Application (Utilities.cs):

[DllImport("DLL", EntryPoint="processImage")]

private static extern short _ProcessQRCode( byte[] in_file,

byte[] out_file );

The process of transferring data through the managed/unmanaged code boundary is known as

marshalling. The .NET Framework is able to automatically marshal C# data types like string into

character arrays. However, C# and the .NET Framework is completely Unicode. Whereas native

applications may or may not support Unicode strings. This was the case with the QR Code reader. The

way around this was to marshal filename paths as arrays of null terminated bytes (See the

`StringToASCIIByteArray` method in Figure 19).


/**

* Convert a Unicode string to an ASCII null terminated

* byte array.

*/

public static byte[] StringToASCIIByteArray(string str)

{

return Encoding.ASCII.GetBytes(str + "\0");

}

/**

* Convert a returned P/Invoke pointer to a null

* terminated array of bytes (chars) to a

* Unicode string.

*/

public static string StringFromASCIIPtr(IntPtr str)

{

List<byte> bytes = new List<byte>();

byte b = 0;

int c = 0;

do

{

b = Marshal.ReadByte(str, c);

bytes.Add(b);

c++;

} while (b != 0);

return Encoding.ASCII.GetString(bytes.ToArray(), 0, bytes.Count);

}

Figure 19 - Marshalling strings to and from unmanaged code

This split code model also creates some difficulties with memory management. Another strength of the

.NET Framework is that it is garbage collected, meaning that the headaches of memory management

are resolved. However, when calling unmanaged code and expecting to receive a pointer to a string

(character array), who is in charge of releasing the memory? The logical approach is to have all memory

managed be performed in native unmanaged code. This approach is used in the QR Code reader

implementation.


iQR Application (UI) Managed C#

Utilities.ProcessQRCode(filename, tempfile);

String filename;

String tempfile;

String decodedMessage;

...

...

Utilities.FreeDLLMemory();

...

decodedMessage = Utilities.GetLastDecodedQRMessage();

iQR DLL Unmanaged C++

processImage(char*,char*)

uchar * getLastMessage()

disposeDLL()

uchar * last_message

Figure 20 - Memory management between managed and unmanaged code

First there is a call into the unmanaged code which passes in the input image filename and the output

image filename. This triggers the decoding process inside the dll which may allocate memory to store

the decoded message in a local dll variable `last_message`. Once the call returns from the dll, this

signals that decoding has completed and if the status is successful, it is safe to query the decoded

message. Lastly we must tell the dll that we have retrieved the entire message and the memory can

now be freed, otherwise we risk introducing memory leaks. Overall this three step process, as

illustrated in Figure 20, ensures that memory is managed in a safe manner.

Ignoring this splitting up of responsibility between managed and unmanaged code, the actual QR Code

detection algorithm and implementation is unchanged from its Windows counterpart (See QR Code

Reader Design section on page 17).

USER INTERFACE

The user interface allows the user to either take a picture of a QR Code or select an existing

image from their library. After an image is selected it proceeds to attempt to decode it. If the image is

able to be decoded, a new screen is displayed with the isolated QR Code, the status of the decode


process, the time it took to decode and the decoded message. If the decoded message is recognized as

a URL, an option appears to navigate to that URL. Below is a collection of the main screens.

Figure 21 - iQR main screen for Windows Mobile

Figure 22 - iQR taking a picture for Windows Mobile

Figure 23 - iQR processing image for Windows Mobile

Figure 24 - iQR decoded message for Windows Mobile


RESULTS

QR CODE READER ACCURACY

Reliably measuring the accuracy of a QR Code reader is difficult because the accuracy is

dependent on many factors such as lighting conditions, camera distance and camera angle. Another

important aspect of determining the accuracy of the QR Code reader algorithm depends on the digital

camera used to capture the image to be decoded. The major drawback of the digital cameras

integrated into mobile devices, is the poor quality of images they produce. Typically they have very

poor light sensitivity, no focus technology, they have a fixed focal-length lens and give the user almost

no control over the capture parameters like shutter speed. For all these reasons there is no

comprehensive test rubric as part of this paper.

However, there are some interesting observations that are worth noting. For example, decreasing the

resolution used to capture images improves processing time but hinders accuracy because blurriness is

harder to correct in lower resolution images. Digital zoom features (like that on the Palm Treo) usually

produce worse images than when brining the camera closer. Typically, the further away a typical

camera is from the QR Code the higher the chance of a successful decode. This is because images

produced by fixed-focal length cameras with no macro image support, produce clearer and sharper

images at about a foot away from the subject, anything higher results in loss of information during the

decoding process.

Both the iPhone and Palm Treo fared well on the test QR Codes found in Appendix B on page 62. The

iPhone was able to successfully decode every QR Code listed from the first try. The Palm Treo had

some difficulties with the small versions of each QR Code, most notably having difficulties with test

sample n0. 3.


QR CODE READER PERFORMANCE

As part of the multi-platform development undertaking of this project, it is important to

compare how nearly identical code and implementations fare on different the devices.

The hardware used for testing the iPhone code was the Apple iPhone 3G running the iPhone 3.0 OS.

The iPhone’s processor has a clock speed of 412MHz and includes on-chip vector floating point (VFP)

support. It also has 128 MB of RAM, however during my testing only 40 MB was actually available to

applications.

The hardware used for testing the Windows Mobile code was the Palm Treo 700wx running Windows

Mobile 6 OS. The Treo’s processor has a clock speed of 312MHz and does not include on-chip floating

point support. Along with a total of available 62 MB of RAM and only about 40 MB was available to the

applications. The lack of floating point number support I suspect put the Treo at a disadvantage.

Figure 25 - QR Code reader performance graph

7 8 8 7 7 8 8 79

4643 44 44

42 4346

4345

0

5

10

15

20

25

30

35

40

45

50

S M L S M L S M L

1 2 3

De

cod

ing

Tim

e (

seco

nd

s)

QR Code Test Sample

QR Code Reader Performance

iPhone WinMo iPhone Average Windows Mobile Average


The QR Codes used in producing the graph (Figure 25) can be found in the Appendix B section on page

62. The two mobile devices were used to take photos of the printed versions of these QR Codes at equal

height distances and under the same lighting conditions. The x-axis is split by QR Code test sample

number and further by the size of the QR Codes (small, medium, large). The y-axis displays the

decoding time in seconds as reported by the QR Code application on each device.

The first and most obvious thing to notice is that the code running on the Palm Treo was on average 5.5

times slower than its counterpart on the iPhone. This, I think, can be at least partially accounted for by

the fact that the iPhone has a 32% more powerful processor as well as Treo’s lack of hardware support

for floating point operations. Since much of libdecodeqr’s and OpenCV’s code relies on floating point

calculations this will have certainty affected the performance.

Some other things of interest to notice are that the QR Code size relative to the image resolution

showed no significantly measurable difference in the overall time of decoding. The amount of encoded

data also had no measurable effect.

Overall, the average time to decode a QR Code on the Palm Treo was 44 seconds, while on the iPhone

it was a mere 8 seconds.


CONCLUSION

This project has been a great learning experience and has resulted in a useful addition to field of

computer vision. All the goals and objectives (See the Goals and Objectives section on page 14) have

been successfully achieved. Three variants of the QR Code reader application have been designed,

developed and tested. Also the process of porting the popular computer vision framework, OpenCV, to

the iPhone and Windows Mobile platforms has been pioneered and documented providing a

foundation for future work. This paper marks the first time that OpenCV has been successfully ported

to the Windows Mobile 6 platform.

This project represents the culmination of a full semester’s work. I believe that this project will serve as

a great foundation for future work in the field of computer vision.

CONTRIBUTIONS TO THE FIELD

This project makes several contributions. The main one is the important research, development

and testing work done in porting the OpenCV computer vision framework to the Windows Mobile

platform. This paper is the first to document the process of porting the OpenCV framework to the

iPhone and Windows Mobile platforms.

Along with the effort of porting OpenCV, a QR Code reader application was developed for the iPhone

and Windows Mobile platforms demonstrating the ideas of this paper. This paper also provides an

overview of the work required in successfully developing multi-platform applications.

I hope that this paper will serve as a guide for future projects based on OpenCV development for mobile

devices.


FUTURE WORK

There is definitely room for a lot of interesting future projects. What follows are some

interesting directions to go from this project.

MonoTouch is an iPhone SDK developed by Novell, making it possible to develop iPhone applications

using the C# language and the .NET Framework (Novell, 2009). This means that platform portability

has been made even easier. In the case of this project for instance, the iPhone version of the QR Code

reader could have been ported over from Windows Mobile, with little changes. Such multi-platform

possibility is very promising and definitely deserves future project consideration.

Augmented Reality (AR) is an up and coming part of computer vision which is seeing a lot of attention

due to the slew of capable mobile devices. Augmented reality is the fusion of a real world video feed

being merged with computer generated imagery (Haller, Billinghurst, & Thomas, 2007). Some

applications rely on tags, similar to the QR Code, for calculating relative camera properties and

inserting generated content. Future work can look into leveraging OpenCV’s augmented reality

capability and creating mobile applications that innovate how augmented reality is applied on mobile

devices

Another interesting project that could further the work accomplished in this paper, is to add the ability

to recognize and decode QR Codes in video. The video can be a live camera feed in which case high-

speed decoding performance would be critical. Also finding and decoding QR Codes in a pre-recorded

video file would have interesting applications.

Finally, something that has not been covered as part of this project but would definitely be of use is to

build a test kit for benchmarking OpenCV on the new platforms that it has been ported to.


REFERENCES

Apple Inc. (2009). iPhone Dev Center. Retrieved November 29, 2009, from iPhone Dev Center:

http://developer.apple.com/iphone/

Apple Inc. (2009, July 28). The Objective-C Programming Language. Retrieved November 28, 2009, from

Objective-C Programming Guide:

http://developer.apple.com/mac/library/documentation/Cocoa/Conceptual/ObjectiveC/ObjC.pdf

Apple iPhone 3G - Full phone specifications. (2009). Retrieved November 25, 2009, from GSM Arena:

http://www.gsmarena.com/apple_iphone_3g-2424.php

ARM architecture. (2009, November 24). Retrieved November 25, 2009, from Wikipedia:

http://en.wikipedia.org/wiki/ARM_architecture

Baijian Yang, P. Z. (2007). Professional Microsoft Smartphone Programming. Wrox Press.

Barcode. (2009, November 18). Retrieved November 20, 2009, from Wikipedia:

http://en.wikipedia.org/wiki/Barcode

Bellis, M. (2009, November 20). Bar Codes - The History of Bar Codes. Retrieved November 20, 2009,

from About.com: http://inventors.about.com/library/inventors/blbar_code.htm

Bradski, G., & Kaehler, A. (2008). What Is OpenCV? In G. Bradski, & A. Kaehler, Learning OpenCV (p. 1).

O’Reilly Media, Inc.,.

Chai, S. (2009). Mobile Challenges for Embedded Computer Vision . In S. Chai, Embedded Computer

Vision (pp. 219-235). Springer London.

CIA - The World Factbook - World - Communications. (2009, November 15). Retrieved November 15,

2009, from CIA - The World Factbook: https://www.cia.gov/library/publications/the-world-

factbook/geos/xx.html

Haller, M., Billinghurst, M., & Thomas, B. (2007). Emerging Technologies of Augmented Reality: Interfaces

and Design. IGI Global.

Hayward, R., & Kim, D. (2009, December 7). Explore a whole new way to window shop, with Google and

your mobile phone. Retrieved December 9, 2009, from Official Google Blog:

http://googleblog.blogspot.com/2009/12/explore-whole-new-way-to-window-shop.html

iPhone. (2009, November 25). Retrieved November 25, 2009, from Wikipedia:

http://en.wikipedia.org/wiki/IPhone

Kairer, R. (2006, October 24). Palm Treo 700wx. Retrieved November 27, 2009, from Palm Infocenter:

http://www.palminfocenter.com/news/9058/palm-treo-700wx-review/


Keserovic, S., Mortenson, D., & Nathan, A. (2003, October). An Overview of Managed/Unmanaged Code

Interoperability. Retrieved November 27, 2009, from Microsoft MSDN: http://msdn.microsoft.com/en-

us/library/ms973872.aspx

Microsoft. (2008, October 16). Windows Mobile 6. Retrieved November 29, 2009, from Windows Mobile

Developer Center: http://msdn.microsoft.com/en-us/library/bb158486.aspx

Novell. (2009). MonoTouch From Novell. Retrieved November 29, 2009, from MonoTouch From Novell:

http://monotouch.net/

Rhonda Ltd. (2009, February 04). OpenCV vs. Apple iPhone. Retrieved November 28, 2009, from

Computer Vision Software: http://www.computer-vision-software.com/blog/2009/04/opencv-vs-apple-

iphone/

Takao, N. (2007). libdecodeqr. Retrieved November 29, 2009, from libdecodeqr: http://trac.koka-

in.org/libdecodeqr

Wheeler, R (Designer). (2008, January 7). QR Code Structure Example. Retrieved November 25, 2009,

from Wikipedia: http://en.wikipedia.org/wiki/File:QR_Code_Structure_Example.svg


APPENDICES

APPENDIX A – CD CONTENTS

Along with a copy of this paper the accompanying CD contains the notes I kept during the

process of cross compiling OpenCV for Windows Mobile. This includes any and all major changes that

had to be done to the code in order to make it compile. What follows is the CD file structure with brief

explanations of the contents.

- Libdecodeqr – QR Code decoder library compiled for the three main platforms discussed.

- iPhone 3.0

- Windows

- Windows Mobile 6.0

- OpenCV-2.0.0 – OpenCV libraries (cv, cxcore, ml, highgui) compiled for the three main

platforms discussed.

- iPhone 3.0

- build_opencv.sh – Bash script used for compiling OpenCV. See section iPhone

on page 24 for more information.

- Windows


- opencv_winmo.patch – Patch diff file used compiling OpenCV. See section

Windows Mobile on page 27 for more information.

- Miscellaneous Notes.pdf – Notes kept during the cross compiling stage of

project.

- QR Code Reader – Contains the iQR applications discussed in this paper and all the associated

code, project and configuration files.


- iPhone 3.0

- Windows


- Alex Karpus - Honours Project 2009.pdf – This paper for reference.


APPENDIX B – TEST CASES

The following QR Code test samples were generated using a QR Code Generator provided by

the team from the ZXing Project.

The QR Code Generator can be found at: http://zxing.appspot.com/generator/

More information about the ZXing project can be found at: http://code.google.com/p/zxing/


QR CODE TEST SAMPLE #1 “http://scs.carleton.ca”


QR CODE TEST SAMPLE #2 “http://en.wikipedia.org/wiki/QR_Code”


QR CODE TEST SAMPLE #3 “an opencv implementation of a qr barcode reader for the iphone and windows mobile platforms”

Alex Karpus Honours Project 2009

Documents

Transcript of Alex Karpus Honours Project 2009