SEIZE THE DATA. 2015

© Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.1 SEIZE THE DATA. 2015

SEIZE THE DATA. 2015


IDOL Rich Media (speech, image, and video) under the hoodDavid Humphrey / Month day, 2015

© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

“The increasing use of multimedia… has contributed significantly to the growth of big data and will continue to do so.”

McKinsey Global Institute


Where does IDOL Rich Media sit?

Ingest Enrich ProduceAnalyze


Speech and audio analytics


Differing approaches

Phonetics search and indexing

Converting audio into raw sound

Pattern matching sounds for results

Word spotting search and indexing

Converting audio into words

Pattern matching words without meaning

Language model based Search & Indexing

Conceptual search

Converting audio into words and concepts

Statistical pattern matching the concepts with understanding


Acoustic model, language model and lexicon for each language

IDOL Speech is supported by powerful algorithms

• 30+ languages supported

• Real-time operation

• Speaker Independent

• Ability to customize language

• Telephony and broadcast models

Models of fundamental sound patterns – different for low quality telephone models (8kHz) and higher quality broadcast models (16Khz+)

Base language models and customized models that include common phrases and word sequences

Trained pronunciationdictionary with

vocabulary

TextFront end processing

Recognition algorithm

Languagemodel

Lexicon

Acousticmodel

Speech


Speech-to-text technology

Statistical models of speech and language

P(W) = probability of word string W

Language model

P(A|W) = probability of a acoustic sequence A given W

Acoustic model

Use Bayes rule to find the word string w that has the highest probability given the acoustic sequence

W = arg max P(W|A) = arg max P(W) P(A|W)P(A)

Language model Acoustic models


Language model

Provides probability of word sequence

Forms a conceptual understanding of language

“Can I help you?”

“Can eye help you?”

Trained from large text corpora (100s of millions of words)

Defines words that can be recognized

Use training text, e.g. broadcast news

Encompasses topic information, colloquial phrases, etc.

Adaptable for particular customer

Specialist vocabulary, e.g. product names


Acoustic model

Model the sounds that comprise a spoken language

Audio analyzed to extract energy at various frequencies

Dependent on audio format

16kHz sampled data for broadcast

8kHz sampled data for telephony

Complex statistical techniques model both the sounds and audio characteristics

State-of-the-art models for co-articulation: “could you” -> “cud ju”


Supported languages

English• American

• British

• Australian

• Canadian

• Singaporean

French• European

• Canadian

German

Spanish• European

• North American

• South American

Italian

Danish

Dutch

Swedish

Flemish

Portuguese• Brazilian

• Portugal

Welsh

Catalan

Polish

Greek

Romanian

Czech

Slovak

Russian

Japanese

Korean

Mandarin

Arabic• Modern Standard

• Gulf

Farsi

Urdu


Other functionality

Speaker ID

Spoken Language Identification

• Phonic sequences

Audio (phonetic) Phrase Search

• Phonic probalistic matches

Speech Segmentation

Transcript Alignment

Audio Classification

• Music, speech silence

Audio Quality

Audio Fingerprinting

• Key frequency points

Audio Security

• Glass, gunshot, scream


Automatically monitor channels in many languages

Challenges with traditional solutions• Rapid increase in broadcast sources

• Exponential growth in number of online video sources

HP Broadcast Monitoring• Fully-automated, real-time, multilingual broadcast news monitoring across multiple sources

• Real-time transcription and translation of all available audio sources, 30+ available languages


Image analytics


Basic terminology

Detection Categorization & classification

Recognition

Description Detecting that a particular type of object is present

Detecting objects of particular categories are in an image or classifying objects by feature

Identification of a particular instance of an object

Example • “There are 3 faces in this image”

• “This face looks middle aged.” • “The faces are Barack Obama, and George P. Burdell”

• “The logo shown here is the Hewlett Packard logo”

• “It looks like the Mona Lisa is in this photo”


Find it. Analyse it.

Video and image analytics



Text OCR

ANPR




Text OCRIDR

ANPR

Barcodes




Text OCRIDR

ANPR

Faces

Person ID

People countDemographics

Barcodes




Text OCRIDR

ANPR

Faces

Person ID

People countClothing colours

Demographics

Barcodes

Colours




Text OCRIDR

ANPR

Faces

Person ID

People countClothing colours

Demographics

BarcodesObject match

Object classObjects

Colours

ISAS



Find it

Building blocks of computer vision

Pre-process

Colour analysis

Grayscale conversion

Corner detection

Edge detection

Prior knowledge & rules

Texture analysis


Find it

Tell-tale signs

y

|Δpx|

High contrast

|Δpx|

x


Find it

Tell-tale signs


OCR analytics


Color


Barcode and Qcode


Face analytics


Object analytics


Image classification

The general problem:

Searching images against a large database for similarity of visual appearance

The specific issue:

What is the exact definition of similar?

Two approaches

Features then SVM

• Faster and easier to train

Neural Nets

• Better performance

• Much harder to train

Image Similarity Ranking

Less similar

Search source


What is similar?

Trade mark infringement

Pornography detection for site blocking

Weapon identification for interactive training

Video scene indexing

Searching

Add insertion

Surveillance object type

Car/van/bus


Scene classification


Crowdsourced video and image classification


Video analytics


Find it. Track it. Monitor it.

Video as a series of stills

Images Find text, faces, paintings, … (anything defined)

Repetition of Image analytics:• Combine results for greater accuracy.

• You found it, now track it.

• Something is there. Still there. Suggest it.

Video Find anything (that moves or moved)• Detect suspicious objects/events

• Improve human efficiency


Video Management System (VMS) Video source• H264, H263, MPEG4, MJPEG, MPEG2

• Analogue, IP, Hybrid

• OnVif, RTSP, RTP/RTCP, UDP

• Unicast, Multicast and File

• MISB

Recording modes

Real-time, time lapse, event driven, VMD and “Lip-sync” Audio

Security

Digital signatures, LDAP mapped

Architecture

On premise, Cloud


Vehicle analysis


License plate recognition


Used in association with ANPR

Vehicle recognition

Match Make and/or Model

• Easy to train

• Real-time matching

Alert or Search for Vehicle without registration

Validate database using ANPR result to identify illegal plated vehicles


ANPR issues


Face analysis


Features• One-to-one verification

• One to many identification

• Retrospective via automatic enrolment

Face recognition and demographics

Demographics

Found “President Obama” face

Body analysis Primary clothing color = whiteNot nude

Primary clothing color = whiteNot nude

Primary clothing color = blackNot nude

Face detection


Use case: Police surveillance


Use case: Counting and access control


Demographic analysis

How many pixels do we need?

Method:• Collect high-res face images

• Rescale each to normalise eyeseparation to 150, 100, etc. pixels

Results:• Best performance is 80%

• Performance drops of a cliff with lessthan 15px between the eyes

Conclusion:NPG need four times resolution to reach the performance plateau

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

020406080100120140160

Acc

ura

cy r

ate

Eye separation/pixels

Combined

NPG data


Scene analysis


Scene analysisUnderstand what is happening in video

Detection of suspicious objects/events• Video motion detect

• Non-motion detect

• Tracking

• Behavioral

• Sequences

• Object identification

Key advantages• Improve human efficiency

• Not limited by human concentration span

• Eliminate staffing issues with 24 x 7 monitoring


Restricted areas


Railways


Traffic


False alarms or missed alarms

Precision = Recall =

False negatives True negatives

False positivesTrue positives

[ Relevant elements ]


Classification of objects

Van - White Person(s)

Car - Silver


Use case: Railway crossing

Business need

• Detect illegal rail crossing activity andautomatically generate evidence pack for court

• Drive down incidents by harsh enforcement

HP solution

• Validate warning light sequence and frequency

• Identify vehicles jumping the lights

• Capture license plate

• Transmit secure evidence pack


Use case: Auckland Transport, New Zealand

Business need

• Improve public safety

• Pedestrians and cyclists

• Over 2000 cameras citywide

• Network of road and environmental sensors

• Real-time social media and news

HP solution

• Initial Phase: Detect high risk activities and investigate threats with scene analysis and licenseplate recognition

• Future Phase: Uncover breaking trends and facilitate incident responses with social media andbroadcast monitoring


Use case: Race analysis

Business need

• Analyze broadcast video feeds in real-time

• Gain intelligence on other teams

• Gain understanding of audience

HP solution

• Identify in car cameras

• Identify teams

• Identify drivers

• Identify who is saying what

• Understand sentiment

http://www.google.com/url?sa=i&rct=j&q=NASCAR+logo&source=images&cd=&cad=rja&docid=UD8cWsQDCZWA8M&tbnid=aPwNfqPaMPH3qM:&ved=0CAUQjRw&url=http://thefamouslogos.com/?p=319&ei=kW2RUf-XH6GoiALwpIHIBA&bvm=bv.46340616,bs.1,d.cGE&psig=AFQjCNHzpaWZBPSUmxaFpQv073IkA36ofg&ust=1368571656959130

http://www.google.com/url?sa=i&rct=j&q=NASCAR+logo&source=images&cd=&cad=rja&docid=UD8cWsQDCZWA8M&tbnid=aPwNfqPaMPH3qM:&ved=0CAUQjRw&url=http://thefamouslogos.com/?p=319&ei=kW2RUf-XH6GoiALwpIHIBA&bvm=bv.46340616,bs.1,d.cGE&psig=AFQjCNHzpaWZBPSUmxaFpQv073IkA36ofg&ust=1368571656959130


Scene analysis: Issues to be aware of


UAV – Drones


Use case: Force protection

Business need

• Detect threats anytime, anywhere by correlatingintelligence from multiple sources in variousforms – audio, video, reports and 3rd partysensors

HP Autonomy ssolution

• Automatically flag anomalies by analyzing feedsfrom aerostat, UAV, towers, and correlating withother events

• Use biometric databases to relay real-timerecognition of facial features and license plates


Small objects and occlusions


Deviation and scaling


UAV


3D for tracking and modelling


Autonomous flight


Oil theft Niger delta: Coffer dams


Media Management and Analytics Platform


Architecture


ProductsHistoric

• Video logger (engine)

• Image server (engine)

• Speech/Audio server (engine)

• Surveillance

• Wittwin (engine)

• ANPR (engine/app)

• ISAS (engine/app)

• Face (engine/app)

• Visor (app)

• ASA (app)

• Broadcast monitoring (app)

Current

• Video server

• Image server

• Speech/Audio server

• Video management and analysis platform

• Broadcast monitoring

Future

• Media server

• Media management and analysis platform

• Haven Analytics Application Framework


HP MMAP architecture

Widgets

Restful API

Analytics Server ………….. Legacy


Application architecture

Application

Analysis screens

HP MMAP

Video screen

Raw dataCircular bufferOn file system

Search widgets

Video

BD Platform

Video server

Big Data Platform

Video playerwidget

TranscriptWidget

Time Line widget

IDOL

Vertica

Dashboard widgets

JBOSS

CFS

Authorisation

OAuth

Identity provider

LDAP

Expose

d A

PI

REST

AP

I

Player sync’s other widgets

URL redirection

Source ID and UTC timeEvent ID

Video TS files (unsecure)

ThumbnailsMetadata

Indexs

Video TS filesMetadata

ACI

HLS playlist request(Can be distributed)

ACIRequest for Metadata

XML

Source ID

Extracted metadata

UTC time

Event ID

RESTFul (internal by widgets)

Request Metadata

Request Thumbnails

RESTFul

Request Source playlist URL

Enriched data

Token

User ID

Permissions

Custom

3rd Party ?

Video

Metadata

Video, Images, Metadata

Control

Data


Analytical serversVideo, speech/audio and image


Video Server

New ACI server for video processing

Extracts metadata from video streams and outputs it to a variety of destinations

Multi-platform:

Linux 64 bits

Windows 64 bits

Unify multiple products• Focused product development

• Reuse common libraries

• Simplifies deployment

Video Server will be Media server

API Layer

VideoAudioImage

Actions

Metadata

Rolling buffer

File / StreamTransform

ESP Output

Analysis Encoder

Engines:

Ingest

Speech/Audio server


Video Server Ingest Engines

Video Server ingests video through an ingest engine

Receives video input and produces demuxed and decoded streams producing image (Image_n) and audio (Audio_lang_n) tracks that can be processed by other engines

Currently we only have a single Ingest Engine:libAV

Virtually can ingest any video formatshttps://libav.org/general.html#Supported-File-Formats-and-Codecs.

[Ingest]

IngestEngine=LibAV

[LibAV]

Type=libav

https://libav.org/general.html#Supported-File-Formats-and-Codecs


Analysis Engines

List of Analysis Engines included in VideoServer 10.8:

Barcode: Detect and read QR codes FaceDetect: Detect faces Demographics: Obtain demographic information such as age, gender, and ethnicity for detected faces FaceRecognize: Run face recognition on detected faces FaceState: Obtain additional information, such as facial expression, about detected faces Keyframe: Identify keyframesNumberPlate: Detect and read license plates on vehicles Object: Recognize known objects in video ObjectClass: Recognize known object classes in video OCR: Run Optical Character Recognition (OCR) SceneAnalysis: Run Intelligent Scene Analysis to identify important events SpeakerID: Identify speakers SpeechToText: Transcribe speech into text


Event Stream Processing Engines

Attempts to identify meaningful events in streams of event data. Examples:

• Detect the occurrence of a particular word in an audio stream.

• Detect when two events occur within a specific time interval; for example, a number plate is detected within sixty seconds of a traffic light changing to red.

Video Server supports multiple ESP engine types:

• Logical:

− And, And Not, AndNotThen, AndThen, Or

− Deduplication

− Filter

• Lua scripting

ESP engines accept any track type. They also accept the output of other ESP engines.


Encoding Engines

Video Server can encode the video it ingests and encode the video to files

List of Encoding Engines included in VideoServer 10.8:

• Imageencoder: Save image records to disk as image files.

• MPEG: Encode ingested video in MPEG format into disk

• Rollingbuffer Encode video to the rolling buffer into disk


Output Engines

• Output engines transform metadata tracks produced by other engines into different formats and send the data to external systems such as IDOL Server, Vertica, …

• List of Output Engines included in VideoServer:

– Directory: Output information to a directory

– Httpserver: Store the latest record and return it in response to a HTTP request

– IDOL: Output information to IDOL Server

– Vertica: Output information to a Vertica database

– XML* Output information to XML File

– RestFul * Output information to web server

• XSL Transformation

• Index by Time or Event


Output Engines

Output engines transform metadata tracks produced by other engines into different formats and send the data to external systems such as IDOL Server, Vertica, …

List of Output Engines included in VideoServer:

Directory: Output information to a directory

Httpserver: Store the latest record and return it in response to a HTTP request

IDOL: Output information to IDOL Server

Vertica: Output information to a Vertica database

XML* Output information to XML File

RestFul * Output information to web server

XSL Transformation

Index by time or event


Widgets


V2 Player V4 Player V6 Player Native HTML 5 Player

Plug-in technology Active-X + GeckoPepper API (chrome

only)Pepper API (chrome

only)-

Install required Yes No No No

Works in future browsers No Yes Yes Yes

UTC timestamps Yes Yes Yes No

Frame accurate Yes Yes Yes I Frame only

Live streams Yes Yes Yes Safari Only

HLS playback No Yes Yes Safari Only

Reverse playback Yes Yes Yes No

RTSP playback Yes No Yes No

UDP playback No No Yes No

Supported Codecs All standard types All standard types All standard types Browser dependent

File playback No Yes Yes Yes

Video Server Compatible No Yes Yes No

Wittwin Compatible Yes No Limited No

Uses browser cache No Yes Yes Yes

Uses browser proxy settings No Yes Yes Yes


HP Video Player

Default Player HTML Template

AngularJS Directives

MediaElement Java Script API

HPBrowser

plugin V ?

Browser Native MediaElement


TimeLine widget for web pages

Navigate by cursor

Mark inMark out

Display thumbnails

Grab a screen shot

Scroll & zoom

Seek by dateand time

Display metadata


Q&A


Useful tips, tricks and gotchas


Garbage in garbage out

Image quality

Focus

Motion blur

Compression

Image field of view

Pixels on target POT matter

Relative motion

- Bad for ID

- Good for Events

Illumination

Day and night

Weather

Glare

- Winter sun

- Car headlights

IR cameras

Not if colour needed

Megapixel cameras

Great for POT

Bad for Bandwidth and CPU


Bandwidth issues

Camera to server

Multicast

- Needs network support

Dual stream

- If camera supports

Reduced frame rate

- Possibly OK if planned

High compression

- Affects analytics

Server to Command and Control Centre (CC3)

Low res proxy

Evidential copy

C&C to users

Intranet

Web access


OtherAlmost every opportunity in video/image analytics is unique in some way, so get sample video!

Sample images/videos, with specific statements of what is to be identified, are essential to qualify an opportunity.

If they have analogue cameras (old) they will not be very good and will make us look bad.

If this is not the customers first attempt, ask the right questions: who and why did they fail? It may just be impossible!

Identify issues early and flag to customer, do not try to overcome.

Use VMware all the time BUT for ease of deployment not resource sharing.

Proprietary compression with certain manufactures is a no go

We support all main manufactures of cameras but if we don’t have one then get us a loan unit as it doesn’t take long.

ANPR for new region may need new format and fonts, we have a process you follow.

Prior knowledge is our friend!


Storage

Grandma to suck eggs!

1024 OS v’s 1000 Disk

Possibly 2 copies

Low res proxy

Evidential

Most recordings are now Full frame rate 24/7

Events just tag

Event driven recordings need pre and post

Storage period

UK legally 30days max, Internationally this varies massively

Evidence can be stored much longer

Business continuity

Raid default

Mirrored and offsite has been known but rare

General

Video standard PAL

Storage type JBOD

Choose Bit rate (MiBits/S) 2.00

Event recording

Channels per event 0

Days of events 0

Events per day 0

Event length Sec. 0

Frames per second 0.0

Event GBytes 0.0

Continuous recording

Channels of continuous 64

Days of continuous 30

Active hours per day 24

Frames per second 25.0

Continuous GBytes 47835.2

Total Storage TBytes 47.8


Backup


Object recognition


Main use case: Logos


Logos: Easy

Advertising monitoring

Future Adds

Broadcast Monitoring

China Daily

Libray indexing

MediaBin

Police

Tatoo ID

ANPR


Logos: Difficult

Too small

Too slow

Solution:

Tracking across video frames to over gaps(as Aurasma)

Algorithm optimisations


Logos we don’t find

Perspective and Skew

Severe optical ‘bloom’

Complex background

Compression artefacts

Small size

Also:

Train for each view

Solid objects that don’t change shape (Animals)

SEIZE THE DATA. 2015

Documents

Transcript of SEIZE THE DATA. 2015