1 © 2012 The MathWorks, Inc.

A Video Analytics Solution for

Discovering Customer Shopping Behavior

in Retail Stores

Authors: Ayushmaan Basu; Harikrishna G. N. Rai; Shreya Barsaiyan; K Sai Deepak; P. Radha Krishna; Lokendra Shastri

Speaker:

Harikrishna Rai

Research Analyst

Infosys Labs, Bangalore

2

Presentation Overview

Why analytics in retail stores?

Requirements & Assumptions

Challenges in Computer vision

Solution overview

SCID design overview

Algorithm overview

Matlab benefits & toolkits used

Solution benefits

3

Why Analytics in Retail Stores ?

Retailers need personalized and location based

services to improve their customers shopping

experience

Retailers & CPG companies need depth knowledge of

customers data – Shopper Behavior, Buying Patterns and Shelf Activity

– Enable Strategic Marketing (product placement)

Customer profiling is key to offer any such customer

engagement solutions

In-store analytics help stores to take better decisions

about how to tailor sales, coupons and other

promotional campaigns

Get insights into how consumers react to various

displays and digital merchandising scenarios

In-store analytics

Shopper path

Stop points

Floor map

analysis

Time spent on products

RFID & WSN solutions

• Expensive as separate hardware need to attached to each cart

• Maintenance is costly

• Chances of damage to these hardware devices

• Poor scalability

4

Solution Requirements & Assumptions

Assumptions

• Typical retail store setup with aisles for shopping

• Wall/rack mounted camera

• Camera fixed at height of 8-10 feet

• Multiple presence of carts

• Customer movement in moderate speed

• Reasonable lighting conditions

Requirements

• Solution to be simple to use and cost effective

• Zero maintainace

• Robust and highly accurate

• Scalable solution without change in store setup

• Provide additional value added services

• Light weight solution with minimal processor requirement

5

Challenges for computer vision approach

Low resolution images

Varied Lighting condition

Shadows

Occlusion

Orientation

Scale

Perspective projection

6

Demo 1 - - Single Cart Scenario

Demo 2 - - Multiple Cart Scenario

7

Solution Overview vid

eo

str

ea

m

field of view(FOV)

X

Y

Z

Unique Shopping Cart ID

(SCID)

camera distance

Detection of SCID

Recognition

Ca

rt ID

& L

oca

tio

n

Em

be

dd

ed

De

vic

e

Shopper Path

8

Configuration details

Camera Specifications

– a maximum of 2 megapixels resolution at 1FPS

Environment (height, lighting, distance)

– height of camera is fixed at around 8 feet

– fluorescent light

– distance of cart from camera can be between 3-20 Feet

SCID/Marker design specifications

– SCID contains a 3x3 rectangular grid of size is 6” * 6” with unique

symbols

– Black border in SCID is used for detection and it is 0.5” thick

Typical Retail Store with Shopping Carts

9

SCID Overview

ID Color Shape Orientation

9 Green Square NA

5 Blue Rectangle 180 deg

0 Red Triangle NA

- - - -

• Symbols are easily differentiable and identifiable with

combination of 3 primary colors

• Symbols have maximum separation in terms of shape such as • Length and Breadth (e.g., just a ratio may suffice)

• Color (a small number of easily distinguishable colors, e.g., red and blue)

• Orientation along major axis (e.g., horizontal and vertical)

• Redundant coding to enable detection even in cases with

occlusion

• SCID code book can support up to 1000 unique carts on a 3x3

grid

Sample SCID = 509

A B C

B C A

C A B

Code book example for decoding ID 509

10

Start

Grab I’th frame from camera

DETECTION

RECOGNITION

Store Cart ID

Entry

I = I+1

Frame I = 1

Algorithm Overview – Detection & Recognition

11

START

Read Cart ID Entry I

Entry I = 1

Entry(I) ==

END?

WAIT

Is Valid Cart?

I = I+1

PLOT PATH

yes

no

no

yes

Algorithm Overview – Tracking

12

Effect

Sample

Occlusion

(Partial row & column)

Occlusion

(Entire row or column )

Strip Removal

Occlusion scenarios

13

Solution highlights

Computer Vision based recognition system adopting a symbolic

recognition system

Simple encoding with high accuracy

Derogatory effects tackled using redundancy and image

enhancement techniques

Considerable ease of implementation with no customer interaction

Minimal hardware requirements (Porting to DSP Processor -

planned)

Low power/low resolution camera

No change in present environment setup

Completely automated

14

Why Matlab for SCID implementation?

Enabled quick prototyping

Simple to use interface

Availability of ready to use advanced image processing

library

Easy for code walkthrough and debugging

Easy to package as a stand alone solution

15

Matlab Toolbox used

Image Processing

Statistics

Compiler

Builder

16

Image Processing Techniques Used

Morphological Operations

Non Linear Filtering

Image Segmentation

Color Invariance Operations

Feature Extraction and Analysis

17

Infosys SCID - Solution Benefits

Cost reduction – Cheaper vision based software solutions as it requires

only low cost cameras

Low maintenance – Do not require frequent service for hardware devices

Scalability – Highly scalable as only camera need to be procured and

installed for increasing the coverage

Versatility – Same camera set up can be used for other purpose like

surveillance and safety etc

Rich Analytics –Analyses shopper behavior, buying patterns, store map,

most visited aisles, time spent on products etc

In-store analytics solution powered by Infosys SCID* in Matlab environment

* patent pending

18

Thanks

Q&A