Transcript of LightSync: Unsynchronized Visual Communication over Screen-Camera Links Wenjun Hu Hao Gu Qifan Pu.
- Slide 1
- LightSync: Unsynchronized Visual Communication over
Screen-Camera Links Wenjun Hu Hao Gu Qifan Pu
- Slide 2
- 2D barcodes everywhere
- Slide 3
- http://www.windowsphone.com/zh-cn (Linking to online
information)
- Slide 4
- A video of barcodes instead Transmitting information (vs
linking) E.g., information on Gibraltarian attractions E.g., movie
show times for nearby theaters Receiver Transmitter
- Slide 5
- Visual communications on phones No need for additional
connectivity Free, directional, lightweight links Optical
software-defined radio Run as apps on commodity devices
- Slide 6
- Many challenges Frame synchronization Poor image quality
Limited computation power on phones Aesthetics
- Slide 7
- In the rest of the talk Frame synchronization challenges
LightSync design Implementation and evaluation
- Slide 8
- FRAME SYNCHRONIZATION CHALLENGES
- Slide 9
- Frame rate mismatch Screen side (TX) 30 fps video playback
Fairly stable Camera side (RX) 15 30 fps in practice Variable
Lighting conditions Device capability API/system factors High rate,
steady Low rate, unsteady
- Slide 10
- Original frame Received frame patterns
- Slide 11
- Original frame Single frame 2-frame mix Mixing pattern varies
by line Received frame patterns
- Slide 12
- 2-frame mix Mixing pattern (& #components) varies by line
Due to Rolling Shutter CMOS camera sensor scans one line (of
pixels) at a time
- Slide 13
- Received frame patterns Original frame Single frame 2-frame mix
3-frame mix Consecutive frames indicating a missing frame Mixing
pattern varies by line
- Slide 14
- Display frame rate C Frame capture rate C/2 C 2C Occasional
missed frames 4-frame mixes 3-frame mix Occasional missed
frames
- Slide 15
- Synchronized communication Display frame rate C Frame capture
rate C/2 C 2C The only decodable pattern Frame synchronized
Under-utilized and inefficient! Carries information But undecodable
and dropped Cannot be used (of worst camera)
- Slide 16
- Unsynchronized communication Display frame rate (fps) 0 30
Works with any display and capture rates Higher camera capture
rate, better performance Supported frame capture rate
(fps)3015
- Slide 17
- Requirements Need to be able to decode mixed frames In-frame
color tracking per line Need to guard against lost frames/lines
Inter-frame erasure coding Essentially a temporal code across
frames Works with any (monochrome) per-frame code
- Slide 18
- Our solution: LightSync Need to be able to decode mixed frames
In-frame grayscale tracking per line Need to guard against lost
frames/lines Inter-frame erasure coding Essentially a temporal code
across frames Works with any (monochrome) per-frame code
- Slide 19
- LIGHTSYNC DEMO
- Slide 20
- LightSync Need to be able to decode mixed frames In-frame
grayscale tracking per line Need to guard against lost frames/lines
Inter-frame erasure coding
- Slide 21
- Per-line tracking: Mixing patterns 3 frames mixing 3 blocks
mixing Bl or w per block 8 mixing possibilities ??? Frame 1 Frame 2
Frame 3 RXTX
- Slide 22
- Per-line tracking: Mixing patterns RX TX Line 1 Line 2 Line 3
Frame 1 Frame 2 Frame 3
- Slide 23
- Per-line tracking: Mixing patterns Trivial cases RX TX Line 1
Line 2 Line 3 Frame 1 Frame 2 Frame 3
- Slide 24
- Per-line tracking: Mixing patterns Easy cases RX TX Line 1 Line
2 Line 3 Frame 1 Frame 2 Frame 3
- Slide 25
- Per-line tracking: Mixing patterns Tricky cases, to be tracked
per line RX TX Line 1 Line 2 Line 3 Frame 1 Frame 2 Frame 3
- Slide 26
- Code area Per-line tracking: Reference blocks Corner Scanning
direction Tracking bar orientation Tracking bars
- Slide 27
- Per-line tracking: Reference blocks Tx frame 1 Tx frame 2 Tx
frame 3 Tx frame 4 Tx frame 5 (repeats) Tracking bar 1 Tracking bar
2 Tracking bar 3 Tracking bar 4 4 reference blocks per line
- Slide 28
- Per-line tracking: Reference blocks Tx frame 1 Tx frame 2 Tx
frame 3 Tx frame 4 Tx frame 5 (repeats) Tracking bar 1 Tracking bar
2 Tracking bar 3 Tracking bar 4
- Slide 29
- Per-line tracking: Reference blocks Tx frame 1 Tx frame 2 Tx
frame 3 Tx frame 4 Tx frame 5 (repeats) Tracking bar 1 Tracking bar
2 Tracking bar 3 Tracking bar 4 Decoding by comparison Reference
colors of mixes
- Slide 30
- LightSync Need to be able to decode mixed frames In-frame
grayscale tracking per line Need to guard against lost frames/lines
Inter-frame erasure coding
- Slide 31
- Inter-frame coding Simple pairwise XOR Group of 3 original
frames Coded frames
- Slide 32
- Inter-frame coding Looping display on screen Coded frames All
original frames Camera stops frame capture after message
decoded
- Slide 33
- PERFORMANCE
- Slide 34
- Implementation Current frame capture rate Inter-frame
intervals
- Slide 35
- Performance Goal recap: Works with any display/capture rate
combinations Metric: Effective frame capture rate Actual throughput
scales with per-frame capacity
- Slide 36
- Schematic performance For a 20fps phone camera
- Slide 37
- Schematic performance For a 20fps phone camera Ideal Peaks at
the camera rate Level at higher display rates Ideal
- Slide 38
- Schematic performance Peaks at ~ half the camera rate For a
20fps phone camera Conventional Drops to 0 at higher display rates
Ideal
- Slide 39
- Schematic performance Conventional LightSync For a 20fps phone
camera Peaks at the camera rate Slight drops at higher display
rates Ideal
- Slide 40
- LightSync performance Nokia Lumia 920 (~30 fps)
- Slide 41
- LightSync performance Nokia Lumia 920 (~30 fps) HTC One X (~20
fps)
- Slide 42
- LightSync performance Nokia Lumia 920 (~30 fps) HTC One X (~20
fps) Huawei Ascend C8812E (~15 fps) Works with any display and
capture rates Higher camera capture rate, better performance
- Slide 43
- Conclusion and outlook LightSync: Unsynchronized visual
communication Primitive for similar scenarios Lots of potential in
the space of visual communication
- Slide 44
- Thank you! wenjun@microsoft.com
- Slide 45
- Related work Temporal barcode design (inter-frame)
Unsynchronized 4D Barcodes Basically slows down display, only for
TX up to 10fps Spatial barcode design (per-frame layout) COBRA:
Optimized for phone-to-phone PixNet: High-end camera,
computationally intensive, offline processing Other work on visible
light communications