Next Generation Intelligent Camera technologies
Jacob JoseProduct ManagerTexas Instruments
Agenda• TI’s investment in Video and Vision markets• Current technologies for intelligent camera markets• What’s coming next?
TI Confidential - NDA Restrictions3
TI has over 30 years of Video & Imaging innovation
TI leads in intelligent video technologies
Military Public Innovators Early adopters Early majority
DM642 Surveillance Processor with DSP
for analytics
2001
Lower Power
Higher Performance
Decreasing Form Factor• Lower system cost
• Affordable software options• Solve specific market needs• Smaller form factors• Reduced power
Thermal Efficiency
Support Evolving Standard
• Cost of ownership reduction• Standardization begins• Productivity maximized• Easy to implement• Standardization begins
TI’s DMVAx-based products in market NOW
Residential Do It YourselfTI DMVAx based
products in market NOW
Critical Facilities
Gaming Retail
Banking
DM6467 Surveillance Processor with 1GHz
C64x DSP for analytics
2005
VLIB, a library of 50+ optimized
kernels available for C64x, C64x+ DSPs
2007
TI announces first video SoC (DMVA1)
with Vision Co-Processor
2009
TI adds HD compression & Tamper
Detect, Motion Detection, People
Counting on DMVA2
2010
First customers in production using DMVA1 & DMVA2
2011
4
TI opens Vision Co-Processor for
programming using the Vision Co-
Processor Library
2013
Various Analytics technology
development
1990 -
5
TI’s DMVA3:A high performance programmable vision coprocessor
• 720MHz ARM® Cortex™-A8 for control heuristics
• Up to 2 megapixel real-time H.264/SVC-T
• Capable of compressing videos at 20 megapixels or higher resolution while running video analytics algorithms
• Customers can port their own algorithms onto the DMVA3
Loitering Detection
Left Object Detection
Object Classification
Animal
TI’s DM8127:Leading solution for Traffic Cameras
• Up to 750 MHz of C674x floating point DSP
• 1GHz ARM Cortex-A8 for control heuristics
• 5 Megapixel Fish-eye correction + perspective correction (bird’s eye view
• TI’s stereo module library to sense depth and increase accuracy by 5x-10x
• Reference design available
6
Left Image Right Image
Leopard Imaging’s LNC-DUALCAM-3Dbased on TMS320DM8127
Depth Map
DMVA3 and DM8127 enables customizable analytics at the edge for mainstream cameras
7
TI’s DaVinci video processors enable megapixel smart analytics
IP
People counting Trip zone Streaming metadata Tamper detection Motion detection
8
Intelligent Motion Detection (IMD)Configuration Example
Intelligent Motion Detection
Add ROI Delete ROI Clear All
Save Cancel Advanced
Enable ROI Label
Region of Interest Settings
IMD Global Settings
Settings Info
Enable IMDDisplay Object Bounding BoxDisplay Enabled Zones
Sensitivity Moderate
Max. Object Size
Min. Object Size 100
8000
Opening page of IMD Control PanelGUI Behavior
For illustrative purposes, the yellow arrow represents an end-
user’s mouse.
This is the opening page for IMD’s GUI, which is called the IMD
Control Panel. It contains a live preview of the camera’s field of
view. ‘IMD Global Settings’, which establish parameters that apply for all regions of interest (ROIs), are
set in the sub-window below preview. ROIs are managed by
the sub-window to the right of the preview window. The lower-right sub-window displays feedback
messages to the user when mouseover events occur or to
acknowledge user input.
When this page is viewed for the first time, we assume that no ROI
has been defined by the user. Note that default settings are
installed and buttons to delete or clear ROIs have been disabled.
Whenever the ‘Defaults’ button is clicked, all settings are restored to
those shown here.
Intelligent motion detection is disabled. Click ‘Enable IMD’ to turn this feature on.
Defaults
Intelligent Motion Detection
Add ROI Delete ROI Clear All
Save Cancel Advanced
Enable ROI Label
Region of Interest Settings
IMD Global Settings
Settings Info
Enable IMDDisplay Object Bounding BoxDisplay Enabled Zones
Sensitivity Moderate
Max. Object Size
Min. Object Size 100
8000
Enabling IMD for the 1st timeGUI Behavior
When a mouseover event occurs over the ‘Enable IMD’ checkbox,
its present status is acknowledged in the Settings Info sub-window.
If the checkbox is clear and the user left-clicks it, report ‘Intelligent
motion detection is enabled.’ in Settings Info. Likewise, if the
checkbox has been checked and the user left-clicks it, report
‘Intelligent motion detection is disabled.’
If the mouse returns to the checkbox (after loosing focus), report ‘Click this button to turn
intelligent motion detection off.’ in Settings Info.
Click this button to turn intelligent motion detection on.
Defaults
Intelligent Motion Detection
Add ROI Delete ROI Clear All
Save Cancel Advanced
Enable ROI Label
Region of Interest Settings
IMD Global Settings
Settings Info
Enable IMDDisplay Object Bounding BoxDisplay Enabled Zones
Sensitivity Moderate
Max. Object Size
Min. Object Size 3
67
Changing default settingsGUI Behavior
The ‘Sensitivity’ dropdown list can be set to the following states:• Low• Low-to-Moderate• Moderate• Moderate-to-High• High
When a mouseover event occurs over the ‘Sensitivity’ dropdown list, instructive information is shared in the Settings Info sub-window: ‘Try
Low sensitivity to improve performance in outdoor or
challenging environments. High-sensitivity works best for fast
moving objects in stable lighting environments.’
The Min. Object Size slider changes the radius of a green circle rendered on the screen. Likewise, Max. Object Size will affect the radius of a red circle.
These circles are drawn anytime a mouseover event occurs on the slider or adjacent input window.
The user is free to move the circles to any location inside of the
preview window.
Move the slider or type in a value to set the minimum object size. Objects that are smaller than this value will be ignored by Intelligent motion detection.
Defaults
Min Obj Size
Max Obj Size
Intelligent Motion Detection
Add ROI Delete ROI Clear All
Save Cancel Advanced
Enable ROI Label
Region of Interest Settings
IMD Global Settings
Settings Info
Enable IMDDisplay Object Bounding BoxDisplay Enabled Zones
Sensitivity Moderate
Max. Object Size
Min. Object Size 100
8000
Enabling with no user-defined ROIsGUI Behavior
If the user clicks ‘Save’ when no user-defined regions of interest have been selected, a DMVA
Settings Manager dialog pops up and displays a message. This
dialog has the focus of attention over any other control owned by the browser window. The dialog
has redundant controls (same as those presented within the IMD
Control Panel).
If the user clicks ‘Save’, a new ROI is automatically defined by a 4-point polygon that shares the same coordinates as the entire frame. This ROI is enabled and labeled as ‘Field_of_View’. The dialog is closed and the page is
redirected to DMVA home.
If the user clicks ‘Cancel’, the dialog is closed and the focus
returns to the IMD Control Panel.Intelligent motion detection is enabled.
Defaults
Since no regions of interest have been explicitly defined, an alert event will be
signaled when any object motion is detected within the entire field of view.
Press Save to accept or Cancel to return to the IMD Control Panel.
DMVA Settings Manager
Save Cancel
Intelligent Motion Detection
Add ROI Delete ROI Clear All
Save Cancel Advanced
Enable ROI Label
ROI_0
Region of Interest Settings
IMD Global Settings
Settings Info
Enable IMDDisplay Object Bounding BoxDisplay Enabled Zones
Sensitivity Moderate
Max. Object Size
Min. Object Size 100
8000
Adding a region of interest (1)GUI Behavior
If a mouseover event occurs over the ‘Add ROI’ button, an
informative message is presented in ‘Settings Info’.
If the user clicks this button, the top row in the scrolling list is
highlighted in a brighter color. In this row, an enabled checkbox appears in the 1st column and a default ROI label, e.g. ROI_0,
appears in the 2nd column.
Click on this button to define a polygon to represent a new region of interest..
Defaults
Intelligent Motion Detection
Add ROI Delete ROI Clear All
Save Cancel Advanced
Enable ROI Label
ROI_0
Region of Interest Settings
IMD Global Settings
Settings Info
Enable IMDDisplay Object Bounding BoxDisplay Enabled Zones
Sensitivity Moderate
Max. Object Size
Min. Object Size 100
8000
Adding a region of interest (2)GUI Behavior
After the ‘Add ROI’ button is clicked, the mouse pointer
changes from arrow to crosshairs when the mouse moves inside preview window. An instructive
message is presented in ‘Settings Info’ that explains how to define
the ROI.
Clicks outside the preview window force a warning message inside Settings Info, e.g. Invalid point selected! Please pick a point inside the preview window.
If 16 points have been defined while setting the ROI, the next single click will complete the
polygon based on those points.
Define the region of interest. Each left mouse button click within the live preview window defines a point in the polygon. Up to 16 points can be used. Double-clicking the left mouse button will automatically complete the polygon.
Defaults
Intelligent Motion Detection
Add ROI Delete ROI Clear All
Save Cancel Advanced
Enable ROI Label
ROI_1
ROI_0
Region of Interest Settings
IMD Global Settings
Settings Info
Enable IMDDisplay Object Bounding BoxDisplay Enabled Zones
Sensitivity Moderate
Max. Object Size
Min. Object Size 100
8000
Adding a region of interest (3)GUI Behavior
After the ROI has been established, the area inside the
lines is shaded and the label displayed. Buttons ‘Delete ROI’ and ‘Clear All’ become enabled and the top row of the ROI list
loses the focus (no highlighting).
To add more ROIs, the user must click ‘Add ROI’ for each region. Once activated, the user must
click the left mouse button inside the preview over a location not
already defined by an existing ROI to create a new ROI. The
elements in the ROI list will grow accordingly.
Define the region of interest. Each left mouse button click within the live preview window defines a point in the polygon. Up to 16 points can be used. Double-clicking the left mouse button will automatically complete the polygon.
Defaults
ROI_0
Intelligent Motion Detection
Add ROI Delete ROI Clear All
Save Cancel Advanced
Enable ROI Label
DESK
ROI_0
Region of Interest Settings
IMD Global Settings
Settings Info
Enable IMDDisplay Object Bounding BoxDisplay Enabled Zones
Sensitivity Moderate
Max. Object Size
Min. Object Size 100
8000
Editing a region of interestGUI Behavior
The user can change the label name at any point by click on the
label in the ROI list. Any name under 64 characters is exceptable
as long as the name is unique, case-insensitive, begins with a letter, etc. (same restrictions as filenames). When editing, the
cursor is displayed and the entire row is highlighted.
If the another ROI was being defined when the user tries to edit a label name, store the settings to
a valid state, e.g. close the polygon as if a double-click were
registered.
Re-label the name of the region of interest. Limit names to 64 characters and avoid leading numbers and the following characters: ` , ~ ! @ # $ % ^ & * ( ) / ? ‘ { } \ |
Defaults
ROI_0
DESK
Intelligent Motion Detection
Add ROI Delete ROI Clear All
Save Cancel Advanced
Enable ROI Label
DOOR
DESK
MAIN_FLOOR
Region of Interest Settings
IMD Global Settings
Settings Info
Enable IMDDisplay Object Bounding BoxDisplay Enabled Zones
Sensitivity Moderate
Max. Object Size
Min. Object Size 100
8000
Deleting a region of interestGUI Behavior
If a mouseover event happens over the ‘Delete ROI’ button, Settings Info reports ‘Click to
delete a highlighted ROI.’
Pressing this button without first selected a ROI label name to do nothing; Settings Info will report
‘You must select a ROI label name first to delete.’
Double-clicking a label name gives it the focus of attention, e.g. highlights it. If ‘Delete ROI’ is
clicked after the row is highlighted, the corresponding overlay is also brightened. Clicking this button a
second time will delete the element in the list as well as the
overlay region in the preview window. If the focus of attention is
diverted to the keyboard or another location on the page not
over the button, the delete operation is canceled.
No overlap tolerated. Max. ROI = 16
Click again to delete a highlighted region of interest.
Defaults
ROI_0
DESK
DOOR
18
DMVA Event Monitor
Excerpts from DMVA Graphical User Interface (GUI) Specification
DMVA Event Monitor
Timestamp EventType Event Details
Tu Oct 7 0913:29:39.1 IMD Zone = LOBBY
Duration = 00:1:39.2
Tu Oct 7 0913:34:14.3 IMD Zone = LOBBY
Duration = 00:0:16.3
Tu Oct 7 0913:38:24.5 IMD Zone = LOBBY
Duration = 00:0:59.5
Tu Oct 7 0913:45:19.0 COUNT Num = 2
Direction = LATERAL
Tu Oct 7 0913:54:35.8
TRIPZONE Direction = Zone 12
Tu Oct 7 0913:55:19.3 IMD Zone = DESK
Duration = 00:19:49.2
Event List
DMVA Message Window
TimestampMetadata Overlay
DMVA Live Event MonitorGUI Behavior
The DMVA Event Monitor page is the main DMVA page, e.g. homepage. It
features a video window, a scrolling list of event summaries, function
indicators/selectors, controls for display and event recording, and a message
window.
The video window shows both live (default) and archived video, which can
be marked up by graphical overlays. The ‘Display’ subwindow allows the user
to select the desired overlay. A row of indicators/selectors sits below.
Indicator “lamp” for an active analytic function glows “green”; Tamper, which runs at all times, can also glow “red”
during active tamper events. Lamps also used to 1 of the 4 analytic functions
that can run simultaneously with tamper. After function is selected,
user can click ‘Configure’ to bring up the dedicated configuration page.
During each event, the compressed audio and/or video, metadata, and
detailed event log file can be saved.
Tamper alerts are displayed in the DMVA Message window. Overall DMVA health and status information is shown
here as well.
Tamper alert: Camera movement detected Tues Oct 7 2009 13:56:09.2!
LOBBYDESK
Tamper
Analytics
DisplayObject Bounding BoxEnabled Zones
Event RecordingVideo StreamAudio StreamMetadata StreamEvent Log
CancelConfigure Advanced
IMD
CountingTripZone
SMOS
DMVA Event Monitor
Timestamp EventType Event Details
Tu Oct 7 0913:29:39.1 IMD Zone = LOBBY
Duration = 00:1:39.2
Tu Oct 7 0913:34:14.3 IMD Zone = LOBBY
Duration = 00:0:16.3
Tu Oct 7 0913:38:24.5 IMD Zone = LOBBY
Duration = 00:0:59.5
Tu Oct 7 0913:45:19.0 COUNT Num = 2
Direction = LATERAL
Tu Oct 7 0913:54:35.8
TRIPZONE Direction = Zone 12
Tu Oct 7 0913:55:19.3 IMD Zone = DESK
Duration = 00:19:49.2
Event List
DMVA Message Window
TimestampMetadata Overlay
DMVA Live Event Monitor
Cllick ‘Advanced’ to fine-tune camera tamper detection parameters.
LOBBYDESK
Tamper
Analytics
DisplayObject Bounding BoxEnabled Zones
Event RecordingVideo StreamAudio StreamMetadata StreamEvent Log
CancelConfigure Advanced
IMD
CountingTripZone
SMOS
GUI BehaviorIf the Advanced button is pushed, a dialog box appears. For now, it will show camera tamper options.
Press Save to accept or Cancel to return to
the DMVA Event Monitor.
Save Cancel
Advanced DMVA Settings
Detect lights on-to-off eventsDetect lights off-to-on events
Camera Tamper Detection
Sensitivity Moderate Reset Sec. 30
DMVA Event Monitor
Timestamp EventType Event Details
Tu Oct 7 0913:29:39.1 IMD Zone = LOBBY
Duration = 00:1:39.2
Tu Oct 7 0913:34:14.3 IMD Zone = LOBBY
Duration = 00:0:16.3
Tu Oct 7 0913:38:24.5 IMD Zone = LOBBY
Duration = 00:0:59.5
Tu Oct 7 0913:45:19.0 COUNT Num = 2
Direction = LATERAL
Tu Oct 7 0913:54:35.8
TRIPZONE Direction = Zone 12
Tu Oct 7 0913:55:19.3 IMD Zone = DESK
Duration = 00:19:49.2
Event List
DMVA Live Event MonitorGUI BehaviorDMVA generates an output
response for every frame of video. For each event, an summary
record is created that establishes the duration of the event and
associates the timestamp, type, and relevant zone information
(context dependent based on the enabled analytic function).
When a user selects an event from the list and ‘Video Stream’
has also been selected (at time of event as well as while user is
interacting) under ‘Event Recording’, the compressed video
of the event is retrieved and replayed in the window. If
‘Metadata Stream’ has also been select, bounding box information is available for display (depends on whether ‘Metadata Overlay’ has
been checked or not).Video and metadata recorded for this event.
LOBBYDESK
Tamper
AnalyticsIMD
CountingTripZone
SMOS
CancelConfigure Advanced
TimestampMetadata Overlay
Object Bounding BoxEnabled Zones
Video StreamAudio StreamMetadata StreamEvent Log
Display Event Recording
DMVA Message Window
Image Signal Processing (ISP) video processing smart plug-ins
22
Video stabilization: Camera shakes due to weather, traffic → degrades video quality
TI’s advanced video stabilization reduces jitter, improves visual quality
Face detection tracking, recognition:For identification, access control, privacy mask
TI’s face detection HW tracks multiple faces (>30) with detection in <20 milliseconds
TI’s Video Noise Filters preserve detail, enhance visual quality, enable low light imaging
Image noise: Degrades low light image quality, details are lost
Problem Solution
JohnSmith
Problem Solution
Image Signal Processing (ISP) video processing smart plug-ins
TI Confidential - NDA Restrictions
23
Lens Distortion, Fish Eye CorrectionFish eye is lenses can get to 360 degree view
TI’s Lens Distortion Correction technology enables to use cheaper fish eye lenses
Auto White Balance, Auto ExposureFor exposure correction and compensate for type of illumination
TI’s H3A hardware provides for Auto Exposure, Auto White Balance and Auto Focus
Security application: Provide clear picture in high dynamic range scenes
TI’s WDR brings details in the shadow & compensatefor low cost sensors
WHAT’S NEXT?
24
What problem we are trying to solve?
• Two eyes help humans to perceive depth. Still, humans are prone to illusions.
Images from: http://www.qianqin.de/2007/10/15/clockwise-or-counterclockwise/
• Most security cameras have only one “eye”, so they cannot easily distinguish between– shadows and a person– two people standing behind each other
• Depth perception can reduce false detections by 5-10x
50 Years of Intelligent Technology Development at TI
• TI started very early– In 1964, TI started developing a guidance system for a laser-guided artillery shell [1], By
1969, the first guidance kits were delivered [2]
• TI’s analytics algorithms are internally developed, robust video and non-video based intelligence technologies enhanced over a long period of time.
• Today, TI is a leading technology supplier to the automotive market where self-driving cars require far more accuracy than for IPCameras.
26
[2] Note: In 1997 TI exited the defense business, selling it’s defense unit to Raytheon
[1] http://www.ti.com/corp/docs/company/history/timeline/defense/1960/docs/65-ti_demos_laser_guidance.htm
Front Camera
Surround View
Smart Rear Camera
Long Range Radar Others
FusionNight VisionMirror Replacement
Time of Flight technology• Time of Flight technology measures
depth based on how soon the modulated pulses reflect back to source
• Works great at night when stereo is ineffective
• More on TOF technology: http://www.ti.com/ww/en/analog/3dtof/index.shtml
Digital Light Processing (DLP) technology• DLP technology relies on structured
lighting to illuminate a grid on the scene and observe the distortions in the grid to perceive depth
• DLP offers higher resolution than TOF technology
• More on DLP technology: http://www.ti.com/solution/3d_machine_vision
Left Hand Electric Drill Metal Screw
Color maps of measured objects
Comparison of 3D Imaging Technologies Time of Flight (ToF) Stereoscopic Vision Fixed Structured Light Programmable Structured
Light (DLP)
Operational Principle IR pulse, measure light transit time
Two 2D sensors emulate human eyes
Single pattern visible or IR illumination, detects distortion
Multiple pattern visible or IR illumination, detects distortion
Point Cloud Generation Direct out of chipset High SW Processing Medium SW Processing SW Processing scales with # of patterns
Latency Low Medium Medium Medium
Active Illumination Yes No Yes Yes – Customizable Spectrum
Low Light Performance Good Weak Good Good
Bright Light Performance Medium GoodMedium / WeakDepends on illumination power
Medium / WeakDepends on illumination power
Power Consumption Medium/HighScales w/ distance Low Medium Medium
Scales with distance
RangeShort to long rangeDepends on laser power & modulation
Mid rangeDepends on spacing between cameras
Very short to mid rangeDepends on illumination power
Very short to mid rangeDepends on illumination power
Resolution QQVGA, QVGA ->Roadmap to VGA Camera Dependent Camera Dependent WVGA to 1080p ->
Roadmap to WQXGA
Depth Accuracymm to cmDepends on resolution of sensor
mm to cmDifficulty with smooth surface mm to cm µm to cm
Scanning Speed FastLimited by sensor speed
MediumLimited by software complexity
FastLimited by camera speed
Fast / MediumLimited by camera speed
Applications Location X X XIdentification X X X XMeasurement & Inspection X X X XBiometrics XAugmented Reality X X X
Top Related