1

Recent Progress in

Image Sensor Communication

Zhengyuan Xu

Optical Wireless Communication and Network Center

University of Science and Technology of China (USTC)

Hefei, China

http://owc.ustc.edu.cn/

2nd International Conference and Exhibition on Visible Light

Communications (ICEVLC2018)

March 16, 2018, Yokohama, Japan

Published by Visible Light Communications Associated at ''2nd International Conference and Exhibition on Visible Light Communications 2018'' on March 16, 2018 in Yokohama, Japan.This is an open access article under a Creative Commons Attribution-NonComercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0). This means that the work must be attributed to the auhtor (By clause), no one can use the work commercially (NC clause), and the work cannot be modified by anyone who re-uses it (ND clause).

2

Key Research Units at USTC

Team: 7 faculty (1-1000plan, 2-young 1000plan),

2 engineers, 2 staff, graduate students (>40)

Space: 1300 sq. meters

Competitive grants: central government, local

government, industry, CAS, university

Optical Wireless Communication & Network Center

http://owc.ustc.edu.cn/

Key Lab of Wireless-Optical Communications, CAS

Team: 38 faculty from 4 colleges (2-1000plan, 2-Changjiang

2-Distinguished Young Scholars, etc)

Deliverables in recent 5 years:

357 papers, 194 patents, 138 projects (4-973, 13-863, 32-Key Projects)

3

Research Topics

Optical Wireless Communication & Network Centerhttp://owc.ustc.edu.cn/

Optical Wireless

Commun/Networking

Comm

Localization

PAT & Sensing

Wireless Big

Data & AI

Channel

LD lighting, LED reception, array design

LOS/NLOS in/outdoor/underwater channel

Signaling, AMC, dynamic sig detection, IC

S/T/F/C/A/P modulation & diversity techniques

Synchronization, beam/power control

MA/routing/scheduling, resource allocation

Hybrid OWC/RF/wired networks

terminal

environment

biological

Analytical/Experimental study

AoA/DoA/T

oA/TDoA

Data mining

Feature mapping

Data-driven

system design

Device Net

LiFi

WiFixG

UV---VL---IR

4

Books/Chapters, Journal and Conference Publications

Achievements

IEEE/OSA Journals and Conferences

IEEE Transactions on Signal Processing

IEEE Journal of Selected Areas in Communications

IEEE Transactions on Wireless Communications

IEEE Transactions on Communications

IEEE/OSA Journal of Lightwave Technology

IEEE/OSA Journal of Optical Communications and Networking

IEEE Wireless Communications

IEEE Photonics Journal

IEEE Communications Letters

OSA Optics Letters

OSA Optics Express

OSA Journal of the Optical Society of America A

OSA Photonics Research

Conferences (IEEE ICC, IEEE GC, OSA CLEO, ECOC)

Book Authors/Editors Year Publisher

1Visible Light Communications:

Modulation and Signal Processing

Authors:

Z. Wang, Q. Wang. H. Huang, and Z. Xu2017 Wiley-IEEE

2

Optical Wireless Communications:

System and Channel Modeling

with MATLAB

Authors:

Z. Ghassemlooy,

W. O. Popoola, and S. Rajbhandari

2017 CRC Press

3Short Range Optical Wireless:

Theory and Applications

Authors:

M. Kavehrad, M.I. S. Chowdhury, and Z.

Zhou

2016 Wiley

4

Principles of LED Light

Communications: Towards

Networked Li-Fi

Authors:

S. Dimitrov and H. Hass2015

Cambridge

University

5Wireless Optical Communication

Systems

Authors:

S. Hranilovic2005 Springer

6Visible Light Communications:

Theory and Applications

Editors:

Z. Ghassemlooy, L. N. Alves, S.

Zvanovec, and M.-A. Khalighi

2017 CRC Press

7Optical Wireless Communications:

An Emerging Technology

Editors:

M. Uysal, C. Capsoni, Z. Ghassemlooy,

A. Boucouvalas, and E. Udvary

2016 Springer

8 Visible Light CommunicationEditor:

S. Arnon2015

Cambridge

University

9Advanced Optical Wireless

Communication Systems

Editors:

S. Arnon, J. R. Barry, G. K.

Karagiannidis, R. Schober, and M. Uysal

2012Cambridge

University

Patents

Journal Editor/Conference Chair (IEEE JSAC’14,’17,

GC OWC Workshop’10-12, ONS’16, IPC’17)

5

Achievements

IEEE Series on Digital &

Mobile Communication

6

Testbeds

Blue OLED

OPD

7

Image Sensors and Characteristics

Image Sensor Communication (ISC)

Background and current status

LED and sensor based ISC noise characteristics and models

Color-intensity modulation in MIMO ISC

LED and phone-camera based cloud ISC

LCD-display and phone-camera based ISC

LED and camera based NLOS scattering ISC

Source Detection and Tracking

Conclusion

Outline

8

Charge-coupled device (CCD)

High-quality, low-noise, high uniformity and sensitivity

high-pixel resolution, but high power and cost

Special need for reconnaissance satellites, spectrometers

biomedical photography, industrial applications

Complementary metal–oxide–semiconductor (CMOS)

Low-power, silicon production line, high

integration, inexpensive, but noisy

Wide applications in smart phones, digital camera

automotive lidar and consumer areas

CMOS manufacturers

Sony, Samsung, Canon, OmniVision, Panasonic, Onsemi,

STMicroelectronics and SK Hynix

CCD vs. CMOS Image Sensors

9

Image Sensor Commun (ISC)

ReceiverTransmitter

Image

Sensor

Industrial

Camera

SLR Camera

Phone

Camera

Webcam

LCD/LED/OLED

DisplayLED Array

Channel

10

A similar example: Quick Response (QR) code

Visible and machine-friendly

Limited information

Limited distance

Covert Display-Phone Communication

Good viewing experience (user-friendly)

Improved information-carrying capability

Longer distance

Enhanced video advertisement

… 011001

LOGO

...

Good Viewing

Experience

Sampling

Decoding Bits

Screen-eye channel

Screen-camera channel

LED Screen

The Embedded Video

The price of this Samsung smartphone is ...

You can purchase from the following website: http://...

LCD-display & Phone-camera based ISC

11

Comparison between VLC and ISC

VLC ISC

Receiver Photodetector (PD) Image sensor /camera

Interference High Low

SNR Low High

MIMO multiplexing Difficult to implement Easy to implement

Decoding Low complexity High complexity

Bluetooth WiFi VLC ISC

Security High High Highest (LOS) Highest (LOS)

Link setup Scan-and-link Scan-and-link LOS Look-and-link

Protocol IEEE 802.15.1 IEEE 802.11a/b IEEE 802.15.7 IEEE 802.15.7m

Frequency band 2.4 GHz 2.4/3.6/5 GHz 400THz-800THz IR, VL, UV

Data rate 800 kpbs 11 Mbps 11kpbs-96Mpbs Lower than VLC

Coverage 30m 46-100m 1m-100m 1m-3km

Image Sensor Commun (ISC)

12

Advantages

Pervasiveness: integrated in most consumer electronics

Massive MIMO receiver: Optical lens+ pixel based receiver

Multi-color receiver: Bayer/Foveon-X3 pattern color filters

Anti-interference receiver: Optical subsystem+ image

processing

Challenges

Unstable frame rate and sampling error

Nonlinearity: O-E conversion of color CMOS sensor

Channel crosstalk: spectrum overlap between LED R/G/B

colors and mismatch with the Bayer-pattern filters

Rolling shutter: readout mechanism of CMOS image sensor

RG RG RG RG RG RG

GB GB GB GB GB GB

RG RG RG RG RG RG

GB GB GB GB GB GB

RG RG RG RG RG RG

GB GB GB GB GB GB

RG RG RG RG RG RG

GB GB GB GB GB GB

Image Sensor Commun (ISC)

13

Research Status

System Affiliation Year Transmitter/Receiver System Performs Feature

[1] PixNet MIT 2010LCD Display, Standalone

Camera7-15Mbps, > 10m 2D OFDM

[2] COBRAMichigan State

U2012 Phone Display, Phone Camera 90-120kbps, 20-30cm 2D Color code

[3] RGB-OCC Northumbria U 2010-2015 LED, SLR Camera 150bps, 60m UPSOOK / MIMO

[4] CIM-OCC USTC 2014-2016 LED Array, Image Sensor 126kbps, 150cm (20m)Color-intensity modulation

(CIM)

[5] NLOS-OCC Feng Chia U 2017 LED, Phone Camera 5.76kbps, 20cm Rolling shutter

[6] OCI-ITS Nagoya U 2014-2017 LED Array, Image Sensor 55Mbps, 30-60m DCO-OFDM

[7] CeilingTalkNanyang Tech

U2017

LED luminaries, Phone

Camera1kbps, 5m Raptor coding

[8] Visual MIMO Rutgers U 2012 LCD Display, Phone Camera 6kbps, BER 0.054Gray images /

Pyramid decomposition

[9] HiLightDartmouth

College2014-2015

Phone Display (60Hz), Phone

Camera1.1kbps, BER 0.1

Color videos / Transparency /

BFSK

[10] InFrame++ Beihang U 2014-2015LCD Display (120Hz), Phone

Camera

150~240kbps, BER

0.1~0.4

Color videos / STCF /

CDMA-like modulation

[11] Spatially

Adaptive Embedding

Rutgers

U/WINLAB2016

LCD Display (120Hz), Phone

Camera22kbps, BER 0.1

Color videos / Block-superpixel

hybrid encoding

[12] IS-based VLC Nagoya U 2016 LCD Display, Image Sensor 1.2kbps, BER< 0.1Color images /

Blue color difference modulation

[13] Invisible

Watermarking

OCC

Kookmin U 2017 LCD Display, Webcam 8bits/image Color images / Cb channel

[14] Covert CSC USTC 2017LCD Display (60Hz)

Phone Camera16.67kbps, BER 0.1

Color videos / Gaussian-based block

design / Synchronization Anchors

14

Research Status

[1] Perli S D, Ahmed N, Katabi D. PixNet: Interference-free wireless links using LCD-camera pairs[C]//Proceedings of the sixteenth annual international

conference on Mobile computing and networking. ACM, 2010: 137-148.

[2] Hao T, Zhou R, Xing G. COBRA: color barcode streaming for smartphone systems[C]//Proceedings of the 10th international conference on Mobile

systems, applications, and services. ACM, 2012: 85-98.

[3] Luo P, Zhang M, Ghassemlooy Z, et al. Experimental demonstration of RGB LED-based optical camera communications[J]. IEEE Photonics Journal,

2015, 7(5): 1-12.

[4] Huang W, Tian P, Xu Z. Design and implementation of a real-time CIM-MIMO optical camera communication system[J]. Optics Express, 2016,

24(21): 24567-24579.

[5] Wang W C, Chow C W, Chen C W, et al. Beacon Jointed Packet Reconstruction Scheme for Mobile-Phone Based Visible Light Communications

Using Rolling Shutter[J]. IEEE Photonics Journal, 2017, 9(6): 1-6.

[6] Yamazato T. V2X communications with an image sensor[J]. Journal of Communications and Information Networks, 2017, 2(4): 65-74.

[7] Yang Y, Hao J, Luo J. CeilingTalk: Lightweight Indoor Broadcast Through LED-Camera Communication[J]. IEEE Transactions on Mobile

Computing, 2017, 16(12): 3308-3319.

[8] Yuan W, Dana K, Ashok A, et al. Dynamic and invisible messaging for visual MIMO[C]//Applications of Computer Vision (WACV), 2012 IEEE

Workshop on. IEEE, 2012: 345-352.

[9] Li T, An C, Xiao X, et al. Real-time screen-camera communication behind any scene[C]//Proceedings of the 13th Annual International Conference on

Mobile Systems, Applications, and Services. ACM, 2015: 197-211.

[10] Wang A, Li Z, Peng C, et al. Inframe++: Achieve simultaneous screen-human viewing and hidden screen-camera communication[C]//Proceedings of

the 13th Annual International Conference on Mobile Systems, Applications, and Services. ACM, 2015: 181-195.

[11] Nguyen V, Tang Y, Ashok A, et al. High-rate flicker-free screen-camera communication with spatially adaptive embedding[C]//Computer

Communications, IEEE INFOCOM 2016-The 35th Annual IEEE International Conference on. IEEE, 2016: 1-9.

[12] Sato S, Okada H, Kobayashi K, et al. Visible light communication systems using blue color difference modulation for digital signage[C]//Personal,

Indoor, and Mobile Radio Communications (PIMRC), 2016 IEEE 27th Annual International Symposium on. IEEE, 2016: 1-6.

[13] Le N T, Jang Y M. Invisible embedded techniques for optical camera communications[C]//Information Networking (ICOIN), 2017 International

Conference on. IEEE, 2017: 599-603.

[14] Wang J, Huang W, Xu Z. Demonstration of a covert camera-screen communication system[C]//Wireless Communications and Mobile Computing

Conference (IWCMC), 2017 13th International. IEEE, 2017: 910-915.

[15] Nguyen T, Islam A, Yamazato T, and Jang Y M. Technical issues on IEEE 802.15.7m image sensor communication standardization[J]. IEEE

Communications Magazine, 2018: 213-218.

15

The M-CIM-MIMO ISC system diagram

LED and Sensor based ISC

Bit Stream

Inserting

Synchronization

Sequence

Inserting

Training

Sequence

CIM

ModulationLED ArrayS/P

Image

Sensor

Array

Anchoring

Symbols

Detection

Training

Sequence

Detection

Frame

Synchronization

CIM

De-modulation

Data

Recovery

FPGA

Control

Bayer-pattern

color filter

Photon

Flux Quantum

Efficiency

Photon

Shot Noise

Photo

Response

Nonuniformty

Dark

Current

Signal

Dark

Current

Shot Noise

Dark Current

Fixed

Pattern Noise

Elec/Volt

Nonlinearity

Sense

Node

Volt/Volt

Nonlinearity

Source

Follow

Source

Follower

Noise

Johnson Noise

(White Noise)

Flicker Noise

(1/f Nose)

Random

Telegraph

Signal Noise

Offset Fixed

Pattern Noise

Correlated

Double

Sampling

ADCIntegral

Linear Error

Digital

Signal

Quantisation

Noise

Photons Electrons

Voltage

Digital Signal

Sense Node

Reset Noise

(KTC Noise)

Frame

synchronization

sequence

Training

signal flag

(previous work)

Training

sequence

(previous work)

Coding area

Positioning

anchor( a )

Header Data

1 Packet

Baker sequence

sM sM sM sM sM s1 s1 sM sM s1 sM s1 sM

(1111100110101)

Preamble 1 Preamble G……

( b )

W. Huang and Z. Xu, “Characteristics and

performance of image sensor communication,”

IEEE Photonics Journal, vol. 9, no. 2, April 2017.

Spatial frame format and

packet format for each LED

Z. Wang, Q. Wang, H. Huang, and Z. Xu, Visible

Light Communications: Modulation and Signal

Processing, Wiley-IEEE Press, NY, Nov. 2017.

Noise

16

CIM-MIMO Modulation and Channel

Considering L-intensity levels and three colors, totally M=L3 CIM symbols

1 2{ , , , }MS s s s [ ] m m m

m r g bs i i i

Channel matrix for N×N Tx-Rx pairs

1,1 1,2 1,

2,1 2,2 2,

,1 ,2 ,

N

N

N N N N

H H H

H H HH

H H H

, , ,

, , ,

, , ,

, , , ,

, , ,

, , ,

i j i j i j

r r r g r b

i j i j i j

i j g r g g g b

i j i j i j

b r b g b b

h h h

h h h

h h h

H

, ,

, ( ) ( ) , , { , , }i j i j j i

p q q ph S F d p q r g b

path loss power spectral density spectral response of filter

, , ,n n n n n k k n

k n

Y X X Z

H H

0 1 2n n nZ Z X Z X Z

Channel input-output relationMixed signal-dependent Gaussian noise:

• photo response non-uniformity (PRNU)

• fixed-pattern noise (FPN)

• random telegraph noise (RTN)

• …

Matrix dimension~1000+

2

2 2 2 2

1 21

n

n n

E XSNR

E X X

17

Parameter Settings

Simulation parameters

Manufacture process CMOS PRNU factor 0.6%~1%

Pixel size 3.75𝜇𝑚 × 3.75𝜇𝑚 Dark current FPN factor 1%

Number of pixels 1080×720 Column offset FPN factor 0.1%

Wavelength 𝛌 550nm Dark current figure of merit 1.00𝑛𝐴/𝑐𝑚2

Fill factor 55% Sense node gain 5.00 𝜇𝑉/𝑒−

Quantum efficiency 65% Read noise 30 𝑒−

Full well 60000 𝑒− ADC bit 12 bit

Experimental parameters

Process CMOS Model OV4688

Pixel size 2𝜇𝑚 × 2𝜇𝑚 Resolution 672 × 380

Data format 10bit RAW Frame rate 330 fps

Binning 4 × 4 Dark current 4mV/sec

18

CIM-MIMO Testbed

Programmable

196-element

RGB-LED

array

maximum resolution 2688*1520FPGA of OCCreceiverRate 330fps

demodulation

256CIM+196-MIMO

Real-time 126kbps,

1.5m, raw BER<1.0e-4,

up to 20m with lens

W. Huang, P. Tian, and Z. Xu, “Design and implementation of a real-time CIM-MIMO optical camera

communication system,” Optics Express, vol. 24, no. 21, October 2016.

19

Noise Distribution

-15 -10 -5 0 5 10 150

0.02

0.04

0.06

0.08

0.1

0.12

0.14

0.16

Received Intensity (mean value substracted)

Pro

ba

bilit

y

Experimental-red

Gaussian fit1

Experimental-green

Gaussian fit2

Experimental-blue

Gaussian fit3

Simulation

Gaussian fit4

0.1539exp(-((x-0.0020)/3.67)2)

0.1524exp(-((x+0.018)/3.69)2)The noise distribution characteristic

The ISC noise is white Gaussian noise

Output noise variance is signal-dependent,

linear function of received intensity, and is

linear function of transmitted signal

0 50 100 150 200 250 3000

0.5

1

1.5

2

2.5

3

3.5

Received Intensity

No

ise

Va

ria

nc

e

Linearity Model

Nonlinearity Model

Experimetnal Measurement

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

x 10-12

0

200

400

600

800

No

ise

Va

ria

nc

e (

Ou

tpu

t G

ray

-va

lue

)

Photocurrent Iph

(A)

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

x 10-12

0

1

2

3

4x 10

6

No

ise

Va

ria

nc

e (

Ou

tpu

t E

lec

tro

ns

e2)

Pixel-sensor model by Konnik

Quadratic fitting (Noise model)

Quadratic fitting (Communication)

M-SDGN Communication Model

quadratic

20

SNR Performance

SNR improves as pixel size, photocurrent,

integral time and diversity order

ISC supports high SNR

Multiplexing/diversity gain tradeoff

10-16

10-15

10-14

10-13

10-12

-40

-30

-20

-10

0

10

20

30

40

Photocurrent Iph

(A)

SN

R (

dB

)

tint

= 1 ms

tint

= 5 ms

tint

= 10 ms

tint

= 20 ms

tint

= 33 ms

tint

= 50 ms

tint

= 100 ms

SNR performance with different parameters

Readout noise

+dark current

dominate

Shot noise

dominates

PRNU

dominates

10-16

10-15

10-14

10-13

10-12

0

10

20

30

40

50

60

70

Photocurrent Iph

(A)

SN

R (

dB

)

B = 1 1

B = 2 2

B = 4 4

B = 8 8

B = 10 10

B = 12 12

B = 14 14

B = 16 16

1010

1011

1012

1013

1014

-20

-10

0

10

20

30

40

50

Incident Photons (1/cm2)

SN

R (

dB

)

1.4 m 1.4 m

1.75 m 1.75 m

2.2 m 2.2 m

2.8 m 2.8 m

3.75 m 3.75 m

5.6 m 5.6 m

6.5 m 6.5 m

7.5 m 7.5 m

21

LED & Phone-camera based Cloud ISC

Array size 16 × 16

LED pixel pitch 1 cm

Array refresh rate 30 fps / 60 fps

Smartphone Samsung note3

Frame rate 60 fps / 120 fps

Resolution 4128×3096

Pixel size 2 um x 2 um

Distance (60-100) cm

Experimental parameters and platform

Data rate 57.6kbps at 1m and BER<𝟏𝟎−𝟑,

viewing angle is up to 40.

Display Cloud

Bits

Inserting frame

synchronization

Inserting

Training

Sequence

CIM

ModulationLED S/P

Channel

Image Sensor

Of

Smartphone

Hough Circle

Detection

Perspective

Correction

Frame

Synchronization

Coordinate

Recovery

Training

Sequence

Detrction

CIM

DemodulationSmartphone

Cloud

22

System Performance

K-means clustering to

optimize constellation

from 64-CIM to 32-

CIM, increasing the

Euclidean distance

BER with different distancesBER with different size of ROI

23

Transceiver diagram

Encoding

Data bits

{ bj }Visual

Modulation

Packeting+

Synchronous anchor

Multiplex

Extract frame

+

Frame rectification

SynchronizationDemodulationDecodingOutput

bits

Video

Sequences Copy / Decompose

Transmitter

Receiver

(1)

(2)

Display

{ Si }

{ S2i-1 }

{ S2i }

{ ej }

{ Dj }

{ Ck }{ Rk }

Intensity modulation

Temporal embedding by Manchester encoding

Spatial embedding by 3×3 Gaussian-based block design

Packet and frame synchronization (anchors)

𝛼 =𝛾

2𝜋𝜎2𝑒−

𝑖−𝑐𝑖2+ 𝑗− 𝑐𝑗

2

2𝜎2

55 pixelpixel

valu

e of

α

1015

2025

30

1015

2025

300

0.01

0.02

0.03

0.04

0.05

0.06

5 10 15 20 25 30

pixel

pix

el

5

10

15

20

25

30

0.005

0.01

0.015

0.02

0.025

0.03

0.035

0.04

0.045

0.05

0.055

1 10, 0 01

LCD-display & Phone-camera based ISC

24

Harris corners

detection

Feature matching between

two sequential Frames (a1,a2)

Frame alignment

a2 to a1

RANSAC

Edge detection

Hough line

detection Four borders

recognization

ROI

extraction

Perspective

Correction

Sobel operatorCaptured frames

Original Video

30fps

Embedded Video

60fps

Captured Video

120fps

1th-group 2th-group

M1 M2

𝑀𝑖 =

1

2𝑓𝑟𝑎𝑚𝑒 1

(𝑖)+ 𝑓𝑟𝑎𝑚𝑒 2

(𝑖)𝑤𝑖,1 = 𝑤𝑖,2 = 1

𝑓𝑟𝑎𝑚𝑒 1(𝑖)

𝑤𝑖,1 = 1 𝑎𝑛𝑑 𝑤𝑖,2 = 0

𝑓𝑟𝑎𝑚𝑒 2(𝑖)

𝑤𝑖,1 = 0 𝑎𝑛𝑑 𝑤𝑖,2 = 1

Perspective correction

Synchronization and demodulation

Determine location of packets by packet anchor

Demodulate frame anchors

Obtain the estimate of two originally transmitted frames

Demodulate information bits block by block

LCD-display & Phone-camera based ISC

25

Experimental Setup

Transmit

ter

27” LCD monitor (AOC G2770PQU)

Brightness 90 %

Refresh rate 60Hz

Resolution 1080× 1920

Receiver

SmartphoneSamsung

Galaxy S5iPhone 8Plus

Frame rate 120fps 120fps

Resolution 720× 1080 1080× 1920

Distance 90 cm (Camera was fixed)Quality assessment

21 volunteers: 16 males + 5 females

8 test videos: the original video + the embedded video

Assessment: score two videos twice separately

0~2 So badly and can not stand

3~4 Strong flicker or artifact

5~6 Evident flicker and unacceptable

7~8 Slight flicker and almost unnoticeable

9~10 No difference and viewing good

If score exceeds 8, we assume that

the quality of the embedded video

is good enough for human eyes.

26

Experimental Performance

Near-zero flicker perception for all the tested videos.

The video quality with chrominance modulation is

better than intensity modulation, especially when video

content is dark (Little Prince) or bright (Sheep).

The average BER with chrominance

modulation is lower than intensity modulation

for all the test videos.

The average BER of chrominance modulation

- iPhone 8Plus is within 0.1.

Communication Performance Rate = 16.67~30.61kbps

Distance = 90cm

Video Quality Scores

J. Wang, W. Huang, and Z. Xu, “Demonstration of a covert camera-screen communication system,” The 13th

International Wireless Communications & Mobile Computing Conference (IWCMC), Spain, June 26-30, 2017.

27

LED & Camera based NLOS Scattering ISC

Non-line of sight (NLOS) ISC

Light source is blocked by vehicle/bldg./tree, etc

Applications in advertising, navigation,

communication link recovery

Challenges in link quality and distance

Camera noise modeling and measurement

HAMAMATSU CMOS C11440-52U

𝑦 = 𝐻𝑥 + 𝑥𝑍2 + 𝑥𝑍1 + 𝑍0

1-5km

Pixels 20482048

Rate 30~25655 fps

A/D 16bit

Digital binning 22, 44

Global exposure 1ms

Noise 2.3 electrons

28

System Performance

Predicted BER vs. range under practical noise

Measurement setup

Obstacle

d

Building1 Building2

Tx Rx

V

LED

AWG

Power

Supply

Driver

Camera

PC

PDFs of measured signal and noise at 130m PDFs of measured signal and noise at 300m

Signal variance increases

with mean value

Maximum distance over

300m at 100bps with error

free

W. Liu and Z. Xu, “Predicted and experimental performance of a long distance non-line of sight image sensor

communication system,” IEEE ICC Workshop on Optical Wireless Communications, Kansas City, MO, USA, May 2018.

29

Source Detection and Tracking

Information

Signal

Tracking

Signal

Light Source

Detection

Motor Control

Signal Recovery

Positioning

Alignment

& TrackingCamera

Lens APD

2-axis Rotating Platform FPGA Board

Image

AcquisitionDifferentiate

Image

ProcessingIdentification

Live

Scene

Coordinate of

light source

Detection

Adaptive region of interest

FPGAPMC

Controller

Servo

Driver

Servo

Motor

Rotating

Platform

APD &

Camera

Tracking

30

Source Detection and Tracking

Camera frame rate 200 fps flag signal freq 100Hz

Resolution 1024×512 Max rotation speed 100°/s

Light source moving fast ~50°/s

31

ISC is a potentially widespread technology in

consumer electronics, and valuable for (covert)

information transfer, PAT, positioning, control

and AI

Display-camera ISC achieved high speed, NLOS

ISC achieved long distance

Challenges still exist in mobility & instability

handling, synchronization, interference

mitigation, capacity increase, computing

capability, etc

Conclusion

32

Zhengyuan(Daniel) Xu

xuzy@ustc.edu.cn

http//owc.ustc.edu.cn