Copyright © 2001 Intel Corporation. On Multimedia Technology & Infrastructure for Emerging...

Copyright © 2001 Intel Corporation.

On Multimedia Technology & On Multimedia Technology & Infrastructure for Emerging Applications:Infrastructure for Emerging Applications:

from Desktop to Wireless to Peer-to-Peerfrom Desktop to Wireless to Peer-to-Peer

Minerva M. Yeung, Ph.D.Minerva M. Yeung, Ph.D.<[email protected]><[email protected]>

Media Technology ResearchMedia Technology Research

Microprocessor Research LabsMicroprocessor Research LabsIntel Corporation.Intel Corporation.

Feb 20, 2001.Feb 20, 2001.Stanford SeminarStanford Seminar

Copyright © 2001 Intel Corporation.Copyright © 2001 Intel Corporation. M. Yeung, 2/20.

IntelMedia Technology

MRL/TRL

Page 2


MRL/TRLAcknowledgements*Acknowledgements* Yen-Kuang Chen, Yen-Kuang Chen, Ph.D.Ph.D. Matthew HollimanMatthew Holliman Rainer Lienhart, Rainer Lienhart, Ph.D.Ph.D. Skip Macy, Skip Macy, Ph.D.Ph.D. Igor Kozintsev, Igor Kozintsev, Ph.DPh.D Andre Zaccarin, Andre Zaccarin, Ph.D.Ph.D. Valery Kuriakin and the iNNL-MPL teamValery Kuriakin and the iNNL-MPL team

(Russia)(Russia) And many othersAnd many others

* Key collaborators/contributors to the work mentioned in the talk. * Key collaborators/contributors to the work mentioned in the talk.



MRL/TRL

Page 3


MRL/TRLUbiquitous MultimediaUbiquitous MultimediaWired and WirelessWired and Wireless

Interactive, High Performance, Interactive, High Performance, Enriched MediaEnriched Media



MRL/TRL

Page 4


MRL/TRLDigital Media: from desktop, to Internet, to Digital Media: from desktop, to Internet, to hand-helds, to wireless, and to Peer-to-Peerhand-helds, to wireless, and to Peer-to-Peer

Edge server

Edge server

Data Farms / Storage

Web ServerApps/DB Server

E-Commerce Server

Media Server

Wireless Comm Server



MRL/TRL

Page 5


MRL/TRL

InequalityInequality: compute, bandwidth, : compute, bandwidth, storage and displaystorage and display

PlatformPlatform CPUCPU MemoryMemory StorageStorage B/WB/W ScreenScreen

ServerServer MultipleMultiple1GHz+1GHz+

2GB2GB 100G to 100G to TerabytesTerabytes

1+ Gbps1+ Gbps N/AN/A

PCPC SingleSingle

1GHz+1GHz+

256M256M 40G40G 100 Mbps100 Mbps 1600 x 12001600 x 1200

LaptopLaptop 600 MHz600 MHz 128M128M 10G10G 100 Mbps100 Mbps 1024 x 7681024 x 768

Media Media

PDAPDA

200MHz200MHz 16M16M 8M8M 19.219.2kbpskbps

320 x 240320 x 240

PDAPDA 70MHz70MHz 8M8M 8M8M 19.2 kbps19.2 kbps 160 x 160160 x 160

Implications:Implications:– Blurring of Core vs. EdgeBlurring of Core vs. Edge

– Migration from client-server to peer-to-peer frameworkMigration from client-server to peer-to-peer framework

– Where should data come from?Where should data come from?

– Where should computations be done?Where should computations be done?

– Optimal partitioning of computationsOptimal partitioning of computations



MRL/TRL

Page 6


MRL/TRLChallengesChallengesWhat is the optimal multimedia What is the optimal multimedia

content distribution mechanism?content distribution mechanism?What is the optimal multimedia What is the optimal multimedia

processing methodology?processing methodology?What are the technology building What are the technology building

blocks?blocks?



MRL/TRL

Page 7


MRL/TRLMultimedia dataMultimedia dataSame content can exist in different Same content can exist in different

resolutions, bit rate and qualityresolutions, bit rate and qualityCan be gracefully degradedCan be gracefully degradedCan be transformed to multiple Can be transformed to multiple

representationsrepresentations



MRL/TRL

Page 8


MRL/TRLP2P for MultimediaP2P for Multimedia Maintain key features and extendMaintain key features and extend

– Supports traditional Supports traditional (existing)(existing) file sharing systems file sharing systems– Facilitates sharing of resources like CPU cycles, storage, Facilitates sharing of resources like CPU cycles, storage,

computation process or arbitrary functionalities that can be computation process or arbitrary functionalities that can be abstracted by the user as resources.abstracted by the user as resources.

Facilitates exchange of data with real time Facilitates exchange of data with real time requirements, exemplified by video or audio requirements, exemplified by video or audio transmission.transmission.

– Choice and flexibility of transport mechanisms Choice and flexibility of transport mechanisms – Meeting diverse application requirementsMeeting diverse application requirements

Efficient multimedia servicesEfficient multimedia services– Integrates media transformation/transcodingIntegrates media transformation/transcoding

Flexible query interfaces to facilitate multimedia Flexible query interfaces to facilitate multimedia searchsearch



MRL/TRL

Page 9


MRL/TRL

MAPS MAPS MAPS

MAPS: Media Accelerating Peer ServicesMAPS: Media Accelerating Peer Services

Technology, platform architecture, and Technology, platform architecture, and software infrastructuresoftware infrastructure– facilitates facilitates efficientefficient, , transparenttransparent and and high-high-

performanceperformance media content delivery & processing media content delivery & processing

– can take advantage of, integrate into, and bring out can take advantage of, integrate into, and bring out the benefits of the P2P computing infrastructurethe benefits of the P2P computing infrastructure



MRL/TRL

Page 10


MRL/TRLSystem ArchitectureSystem Architecture

Network Transport Services (TCP socket, CORBA, DCOM, SOAP)

Peer-to-Peer Service Layer (general security, Directory service, data replication, flow control, etc.)

Multi-media applicationSupport modules (Transcoding, compression,security/watermarking, etc.)

Script language

Application

operating system

applicationTraditional

Support modules of other types of applications (Document editing, distributed processing, etc.)

(Multimedia)MediaAcceleratingPeerServiceLayer

MAPS API

PlatformOptimization

MAPS API



MRL/TRL

Page 11


MRL/TRLSystem ComponentsSystem ComponentsPlatform Optimization (Media Acceleration)Platform Optimization (Media Acceleration)

– Detect and analyze the optimal platform configuration and Detect and analyze the optimal platform configuration and operation at a particular time instantoperation at a particular time instant

– Dynamic optimization via cost-function analysis across a Dynamic optimization via cost-function analysis across a globalglobal or or local local set of peer nodes: the result determines a set of peer nodes: the result determines a bestbest operational mode for a particular media delivery or operational mode for a particular media delivery or processing operationprocessing operation

– OptimizationOptimization is built upon (and calling) a set of support is built upon (and calling) a set of support modules, like streaming, transcoding, etc. for multimedia modules, like streaming, transcoding, etc. for multimedia delivery operations, and other support modules like delivery operations, and other support modules like distributed image processing primitives for multimedia distributed image processing primitives for multimedia processing operations.processing operations.

– InvisibleInvisible and and seamlessseamless to the applications to the applications– Overall results cascaded into an enhanced service Overall results cascaded into an enhanced service

experience experience – fast delivery, higher satisfaction, lower abort rates, and better fast delivery, higher satisfaction, lower abort rates, and better

viewing/listening/interactive experiences, …viewing/listening/interactive experiences, …



MRL/TRL

Page 12


MRL/TRLMinimal Cost to Deliver a Minimal Cost to Deliver a Resource to a Peer NodeResource to a Peer NodeTo find the minimal ofTo find the minimal of

– cost of getting a copy locally and transcoding if cost of getting a copy locally and transcoding if necessary, necessary,

– cost of getting a copy from a peer node + cost of cost of getting a copy from a peer node + cost of transmissiontransmission

Minimal cost Minimal cost Minimal cost

Minimal cost Minimal cost

Minimal cost

i j k

m n



MRL/TRL

Page 13


MRL/TRLCost AnalysisCost Analysis Request to evaluate cost Request to evaluate cost

of Rof R If the file is not cached If the file is not cached

(local), then find the (local), then find the minimal cost from the set minimal cost from the set of neighbors (can extend of neighbors (can extend from local to global)from local to global)

Cost of transcoding Cost of transcoding computationcomputation

Cost of transmission Cost of transmission available network available network bandwidthbandwidth

Other cost factors can be Other cost factors can be formulatedformulated

Cost of retrieval,

transcoding, &transmission

Repeat for the set ofneighbors

Compare the minimal cost

Cache the result

Return



MRL/TRL

Page 14


MRL/TRLMinimal Cost to Perform Minimal Cost to Perform a computationa computationFor a computation task on data (resource R), For a computation task on data (resource R),

optimize with respect to several variables optimize with respect to several variables such as delay, power consumption, temp such as delay, power consumption, temp storage, etc. (constraint parameters P)storage, etc. (constraint parameters P)

To find the minimal ofTo find the minimal of– cost of performing the computation locally, cost of performing the computation locally, – cost of transmitting the data to a peer (or multiple cost of transmitting the data to a peer (or multiple

peers), perform the computation there and peers), perform the computation there and transmitting the result backtransmitting the result back

– cost of finding the same data on a peer, perform cost of finding the same data on a peer, perform the computation there and transmitting the result the computation there and transmitting the result backback



MRL/TRL

Page 15


MRL/TRL

Cost Computation & EstimationCost Computation & Estimation

What is the cost?What is the cost?–Cost should be dynamicCost should be dynamic

– Resource availability at “server”Resource availability at “server”– Resource availability at “client”Resource availability at “client”– Network availabilityNetwork availability

How to measure?How to measure?How to estimate?How to estimate?Methodology?Methodology?Tools?Tools?



MRL/TRL

Page 16


MRL/TRLMethodology for Estimating End-to-End Computing Performance and Analyzing Network Computing Models

Methodology for Estimating End-to-End Computing Performance and Analyzing Network Computing Models

In a networked & distributed computing environment, end-to-end performance of applications depend on capability of components in the system and their interaction.

Need methodology to estimate end-to-end system performance.

Results of performance estimation used to make computing decisions.

– Best distribution of computational tasks.– Best algorithm choice.– Most effective hardware improvement.

– Enhanced security



MRL/TRL

Page 17


MRL/TRLOverviewOverview

results

best distribution

bestalgorithm

lowestcost

models

queuingnetwork

calculations

operations

grayscale

scale size

compression

halftone

encryption

digitalsignatures

Methodology

inputs

processing rates

data size

network bandwidth

cost weights

…

performancemetrics

response time

maximumthroughput

cost

quality

security

Sample Operations



MRL/TRL

Page 18


MRL/TRLSample Results: Computation DistributionSample Results: Computation Distribution

Assumptions: Assumptions: – server processing rate >> client server processing rate >> client

processing rateprocessing rate

– Simplest model; other server/client Simplest model; other server/client disparities not includeddisparities not included

Figure 1Figure 1 Server: grayscale, scale, halftone Server: grayscale, scale, halftone Client: none Client: none Results:Results:

– Processing rate limited. Processing rate limited.

– Lowest response time.Lowest response time.

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

0 4 8 12

arrival rate (requests/sec)

resp

onse

tim

e (s

ec) server process

server network

client network

client process

total

Color convert 512x512 to gray, scale to 128x128, halftone to 1-bit/pixel

0

0.2

0.4

0.6

0.8

1

1.2

1.4

0 2 4 6 8 10

arrival rate (requests/sec)

resp

onse

tim

e (s

ec)

0

0.5

1

1.5

2

2.5

3

0 3 6 9 12 15

arrival rate (requests/sec)re

spon

se ti

me

(sec

)

Figure 2 (Left)Figure 2 (Left) Server: grayscale, scale Server: grayscale, scale Client: halftone Client: halftone

Results:Results:

– Highest request throughputHighest request throughput

– client process raises timeclient process raises time

Figure 3 (Right)Figure 3 (Right) Server: grayscale, Server: grayscale,

scale, compress scale, compress Client: decompress, Client: decompress,

halftone halftone

Results: Results: Bandwidth rate limited. Bandwidth rate limited.



MRL/TRL

Page 19


MRL/TRLSample Results: Quality as a Performance MetricSample Results: Quality as a Performance Metric Compare JPEG2000, SPIHT and JPEGCompare JPEG2000, SPIHT and JPEG

– Order of processing rate: JPEG, SPIHT, JPEG2000 Order of processing rate: JPEG, SPIHT, JPEG2000 – Order of compression ratio for given PSNR: JPEG2000, SPIHT, JPEG Order of compression ratio for given PSNR: JPEG2000, SPIHT, JPEG

ServerServer and and Client Client can have equal capability (measured on PIII 600 MHz systems for the following) can have equal capability (measured on PIII 600 MHz systems for the following)

0

10

20

30

40

50

26 27 28 29 30 31 32 33 34 35 36 37 38 39 40

PSNR

co

mp

res

sio

n

JPEG2000

JPEG

SPIHT

Compression vs PSNR 128x128 24-bit Lena

JPEG2000 2 3

JPEG l6 15

SPIHT 12 12

MethodCompression

(Mb/s) Decompression

(Mb/s) PSNR

30.9

27.8

29.0

Comparison of processing rates

0

0.5

1

1.5

2

2.5

3

26 28 30 32 34 36 38 40

PSNR

maxim

um

th

rou

gh

pu

t

JPEG2000

JPEG

SPIHT

Network limits throughput deceases with PSNR

Processing limited

Maximum throughput for 1.5 Mbps

0

2

4

6

8

26 28 30 32 34 36 38 40

PSNR

dela

y (

sec)

JPEG2000

JPEG

SPIHT

Delay for low arrival rate against PSNR

@1.5 Mbps. JPEG2000 closer

to JPEG and SPIHT for slower

network

Delay increases with PSNR



MRL/TRL

Page 20


MRL/TRLMultimedia SupportMultimedia Support ““Malleability”/“flexibility” of (bulky) MultimediaMalleability”/“flexibility” of (bulky) Multimedia

– Graceful degradationGraceful degradation– Various representationsVarious representations– Complexity Complexity b/w and compute intensive! b/w and compute intensive!– More flexibility More flexibility more opportunities more opportunities

Special technology components in MAPSSpecial technology components in MAPS– Universal across multiple platformsUniversal across multiple platforms– Media Adaptive TranscodingMedia Adaptive Transcoding

– e.g. MPEG2 (6Mbps) e.g. MPEG2 (6Mbps) MPEG4 (300kps) MPEG4 (300kps)

– Audio/video Streaming Audio/video Streaming – Robust, error resilient coding and transmissionRobust, error resilient coding and transmission– Enhanced Multimedia ProcessingEnhanced Multimedia Processing– Technology modules for media security & tracking Technology modules for media security & tracking

(encryption/watermarking)(encryption/watermarking)– Technology that facilitates multimedia browsing/search/managementTechnology that facilitates multimedia browsing/search/management

Intel® MPL can serve as a basic media software infrastructure that provides some core media technology components while facilitates the building of the other ones

Intel® MPL can serve as a basic media software infrastructure that provides some core media technology components while facilitates the building of the other ones



MRL/TRL

Page 21


MRL/TRLIntel MPL - Media Processing LibraryIntel MPL - Media Processing Library(presented at IDF Fall’99, Fall’00)(presented at IDF Fall’99, Fall’00) General purpose, high performance software infrastructure General purpose, high performance software infrastructure

with API’s for MPEG encode/decode and with API’s for MPEG encode/decode and processingprocessing BUILDING BLOCK enabling NEW rich-media apps: targeted for BUILDING BLOCK enabling NEW rich-media apps: targeted for

applications beyond standard encode/decode/display, e.g., applications beyond standard encode/decode/display, e.g., ease-of-use of video, allow interactivity with digital videoease-of-use of video, allow interactivity with digital video

Universal & Ubiquitous:Universal & Ubiquitous:– Common API’s across multiple OS: Windows, LinuxCommon API’s across multiple OS: Windows, Linux– Optimized for multiple architecture: IA, XScale™ Optimized for multiple architecture: IA, XScale™ (in progress)– Can extend to network applications (Can extend to network applications (Internet, wireless, etc.)Internet, wireless, etc.)

Scalable solutionScalable solution– Low bit rate to high bit rateLow bit rate to high bit rate– Low resolution to high resolutionLow resolution to high resolution– 1 processor to multiple processors1 processor to multiple processors– Small devices to high end serversSmall devices to high end servers

Rich features, high performance, ease-of-developmentRich features, high performance, ease-of-development



MRL/TRL

Page 22


MRL/TRLMPL – MPL – PC and Internet Multimedia S/W PC and Internet Multimedia S/W infrastructure for Diverse Platformsinfrastructure for Diverse Platforms



MRL/TRL

Page 23


MRL/TRLReference: MPL featuresReference: MPL features Support MPEG-1 and MPEG-2 up to HDTV, MPEG4Support MPEG-1 and MPEG-2 up to HDTV, MPEG4 Optimized with Optimized with MMX™ technology and SIMD Streaming Extensions, Pentium™ 4MMX™ technology and SIMD Streaming Extensions, Pentium™ 4 Advanced Features - the highlights:Advanced Features - the highlights:

– Random access to any frames with near constant-timeRandom access to any frames with near constant-time– Fast extraction of reduced framesFast extraction of reduced frames– Simultaneous decode of multiple MPEG sequences. Simultaneous decode of multiple MPEG sequences. – Access via callbacks (API) to non-frame level information in the MPEG bitstream, Access via callbacks (API) to non-frame level information in the MPEG bitstream,

such as raw bits, blocks, macroblocks, GOP and slice, etc. [structured access to such as raw bits, blocks, macroblocks, GOP and slice, etc. [structured access to compressed video/audio]compressed video/audio]

– Flexible input plugins: MPL - DTV, MPL – NetworkFlexible input plugins: MPL - DTV, MPL – Network– Audio support (AC3, audio level 1, 2 and 3 – MP3, AAC)Audio support (AC3, audio level 1, 2 and 3 – MP3, AAC)– SMP support; multithreaded designSMP support; multithreaded design– Transcoding capabilitiesTranscoding capabilities

Ease-of-development and deploymentEase-of-development and deployment– Windows*, Linux* Windows*, Linux* – MPL as Component: COM objects (for VB, VC++, DirectShow*)MPL as Component: COM objects (for VB, VC++, DirectShow*)– IPP support for audio & video (2001)IPP support for audio & video (2001)

– cross-platform optimization: Pentium™ processor families, Xscale™ core cross-platform optimization: Pentium™ processor families, Xscale™ core architecture based processors (in planning)architecture based processors (in planning)

* Other names and brands are property of their respective owners* Other names and brands are property of their respective owners



MRL/TRL

Page 24


MRL/TRL

Video processing

control

MixerMixerMixerMixer

Output systemOutput systemOutput systemOutput system

Inside the MPL pipelineInside the MPL pipeline

Encoded stream

Videooutput

Users may easily plug in their own algorithms to encoder’s pipeline and, also, fully control encoding and decoding processes.

Variousinput streams from various devices

PreconditioningPreconditioningPreconditioningPreconditioning

DCTDCTDCTDCT

Control algorithmControl algorithm

Motion estimationMotion estimationMotion estimationMotion estimation

Huffman codingHuffman codingHuffman codingHuffman coding

Headers generationHeaders generationHeaders generationHeaders generation

QuantizationQuantizationQuantizationQuantization

Motion vector search algorithmMotion vector

search algorithm

Bitrate controlBitrate control

Enc

oder

use

r ca

llba

ck f

unct

ion

Enc

oder

use

r ca

llba

ck f

unct

ion

Enc

oder

use

r ca

llba

ck f

unct

ion

Enc

oder

use

r ca

llba

ck f

unct

ion

MPEG encoderMPEG encoder

Video encoding control

Huffman decodingHuffman decodingHuffman decodingHuffman decoding

Headers processingHeaders processingHeaders processingHeaders processing

DequantizationDequantizationDequantizationDequantization

Inverse DCTInverse DCTInverse DCTInverse DCT

Motion CompensationMotion CompensationMotion CompensationMotion Compensation

Dec

oder

use

r ca

llba

ck f

unct

ion

Dec

oder

use

r ca

llba

ck f

unct

ion

Dec

oder

use

r ca

llba

ck f

unct

ion

Dec

oder

use

r ca

llba

ck f

unct

ion

MPEG decoderMPEG decoder

Media inputMedia inputMedia inputMedia input

SplitterSplitterSplitterSplitter

Input systemInput systemInput systemInput system

Comparison of MPL decoding (MPEG-2 704x480)

126 128,2 136,2

0

50

100

150

Tbird (1GHz) PIII (1GHz) P4 (1.1 GHz)Dec

odin

g ra

te

(FP

S)

High performance performance Media Infrastructure



MRL/TRL

Page 25


MRL/TRL

Decoder A

MPL Next StepsMPL Next Steps

NETWORK

Decoder B

Decoder C High bitrate, excellent quality

Average bitrate, good quality

Low bitrate, acceptable quality

Core BitstreamEnhancement Bitstream 1Enhancement Bitstream 2

Demultiplex

Elementary sreams Scenes

decoding

Objectsdecoding

Scene Graph Scene Graph

A/V object 1 A/V object 1 A/V object 2 A/V object 2 Person Person 2D background2D background FurnitureFurniture

Voice Voice SpriteSprite GlobeGlobe DeskDesk

Natural video

2D-Mesh 3D-Mesh Face&Body animation

Synthetic video

BIFS

MPEG-4 (simple to advanced profiles)MPEG-4 (simple to advanced profiles)



MRL/TRL

Page 26


MRL/TRLMPL Next StepsMPL Next Steps Extend to embrace & integrate more technology Extend to embrace & integrate more technology

building blocksbuilding blocks– wirelesswireless

– P2P – embrace technology catering to and take advantage of P2P – embrace technology catering to and take advantage of the “inequalities” inherent in the computing platformsthe “inequalities” inherent in the computing platforms

– low power considerationslow power considerations

– Merging of video & graphicsMerging of video & graphics

Strive toStrive to– Cater to diversity (universal)Cater to diversity (universal)

– Be among the best in Be among the best in performanceperformance

– Be flexible, and maximize utility/usabilityBe flexible, and maximize utility/usability

– Be a key building block infrastructure for emerging Be a key building block infrastructure for emerging applicationsapplications



MRL/TRL

Page 27


MRL/TRL

Facilitating Multimedia SearchFacilitating Multimedia SearchAutomatic media content analysis Automatic media content analysis

algorithms AND meta-data can improve algorithms AND meta-data can improve search efficiency.search efficiency.

New browsing tools and methodologies can New browsing tools and methodologies can be deployed for distributed media sharing, be deployed for distributed media sharing, storage and retrieval, and collaboration.storage and retrieval, and collaboration.

Flexible query interfaces to facilitate Flexible query interfaces to facilitate multimedia searchmultimedia search

Search can be based on BOTH the file Search can be based on BOTH the file names and content descriptions.names and content descriptions.– e.g. MPEG7 or XMLe.g. MPEG7 or XML



MRL/TRL

Page 28


MRL/TRLContent analysis & SearchContent analysis & SearchExample: Video SearchExample: Video Search

– Very compact representationsVery compact representations– Fast Matching on Compact Fast Matching on Compact

SignaturesSignatures

Matched ClipsMatched Clips

representation: 4.26 real numbers per frame of video*

250 min of video searched within 7 sec on a PII 400 against 50 frame query

* achieved using window size 30 and polynomial degree 1;histograms of Y, U, V channels quantized into 16, 8 and 8 bins



MRL/TRL

Page 29


MRL/TRLOn Compact Representations & Fast SearchOn Compact Representations & Fast Search(Naphade, Yeo & Yeung, earlier work)(Naphade, Yeo & Yeung, earlier work) Extract compressed low-resolution temporal signature Extract compressed low-resolution temporal signature

– Compute spatial low resolution signature - for MPEG video, extract/estimate Compute spatial low resolution signature - for MPEG video, extract/estimate DC images for I/P/B frames, on compressed dataDC images for I/P/B frames, on compressed data

– Fit polynomial of N degrees over M samples in timeFit polynomial of N degrees over M samples in time– Describe temporal trace using the N degree polynomial for each M samplesDescribe temporal trace using the N degree polynomial for each M samples

Achieved matching rate of 6000+ frames/sec on a low-end PCAchieved matching rate of 6000+ frames/sec on a low-end PC Reduce complex MM search/matching to a fast string-matchingReduce complex MM search/matching to a fast string-matching Same signature & compact size regardless of resolution of videoSame signature & compact size regardless of resolution of video

DC Sequence Histogram Sequence

Database of Polynomial Coefficients

Similarity MetricComputation

DC SequenceHistogram Sequence

Approximated Histogram Sequence

Incoming Query

Retrieved Clips

Incoming Video



MRL/TRL

Page 30


MRL/TRL



MRL/TRL

Page 31


MRL/TRLBuilding MAPS into a P2P Building MAPS into a P2P computing platformcomputing platform The enhanced multimedia peer service layer can be The enhanced multimedia peer service layer can be

integrated into existing/emerging P2P integrated into existing/emerging P2P infrastructureinfrastructure

– To make P2P media-aware and optimized for media delivery and media applications…

APIs can be defined on P2P platform to facilitate APIs can be defined on P2P platform to facilitate multimedia applications developmentmultimedia applications development

Prototype & experimentation at Intel MRLPrototype & experimentation at Intel MRL

Internet Service “edge” service

Peer-to-Peer Service Media Accelerating

Peer Service MAPS

Internet Service “edge” service

Peer-to-Peer Service Media Accelerating

Peer Service MAPS



MRL/TRL

Page 32


MRL/TRLPrototype systemPrototype system

DaemonDaemon– Low level Low level

primitivesprimitives

– High level High level controllerscontrollers

Client APIsClient APIs – Application Application

access to P2P access to P2P networknetwork

P2P Daemon

Client API

ApplicationsP2P Daemon

Client API

Applications

Peer

Peer

Other Peer

Other Peer

Other Peer

Internet

The Global NetworkUniverse

(Intel MRL 2001)

Peer -to-Peer -to- peer “super” daemon

Daemon -to -daemoncommunication

Cost evaluation module

Media transcodingmodule

Application -to-daemon APIs

Daemon -to -daemoncommunication

Cost evaluation module

Media transcodingmodule

Application -to-daemon APIs

User defined modules



MRL/TRL

Page 33


MRL/TRLSeamlessSeamless Multimedia for Dynamic and Multimedia for Dynamic and Heterogeneous EnvironmentHeterogeneous Environment

HUB802.11b/BT/??

802.11b/BT/??

802.11b/BT/?? 10/100baseT

10/100baseT

Tablets

PDA’s & devices

• Transcoding for bandwidth management• Robust Wireless Streaming/transmission• Multi-platform Encode/Decode• Resource Optimization



MRL/TRL

Page 34


MRL/TRLPeer Media ExplorerPeer Media Explorer

SnapshotSnapshot



MRL/TRL

Page 35


MRL/TRLExtending Multimedia Apps and Extending Multimedia Apps and Technology from Desktop to P2PTechnology from Desktop to P2P

Emerging media-centric applications:Emerging media-centric applications: Content Transaction, Tracking and MonitoringContent Transaction, Tracking and Monitoring Media content indexing, browsing & organizationMedia content indexing, browsing & organization Interactive video, and merged video and graphicsInteractive video, and merged video and graphics Enhanced virtual realityEnhanced virtual reality Enhanced video conferencingEnhanced video conferencing Networked multimedia gamesNetworked multimedia games

Goal: to enable multimedia apps development and deployment on P2P with enhanced user experience.Goal: to enable multimedia apps development and deployment on P2P with enhanced user experience.



MRL/TRL

Page 36


MRL/TRLStreaming, sharing and Streaming, sharing and collaborating on digital mediacollaborating on digital media



MRL/TRL

Page 37


MRL/TRLBuilding BlocksBuilding BlocksBuilding BlocksBuilding Blocks

MPL Call-back/hooks

Ease-of-use as a natural data typeEase-of-use as a natural data type

Media DeliveryMedia DeliveryInteractive VideoInteractive Video

Manipulation/processingfast, intuitive, on un/compressed media, “Word for Video”

Scalable streamingFGSScalable streamingFGS

“Rendering” of video, graphics (& audio)

“Rendering” of video, graphics (& audio)

Automatic content analysis & abstractionAutomatic content analysis & abstraction

Robust/resilient delivery theory and practiceRobust/resilient delivery theory and practice

Optimization of interactive streaming

Optimization of interactive streaming

Search, browse, and retrieveSearch, browse, and retrieve

TranscodingB/W-compute adaptabilitySmarter delivery schemes

TranscodingB/W-compute adaptabilitySmarter delivery schemes

Object recognition & segmentationObject recognition & segmentation

WatermarkingEncryption, SecurityWatermarkingEncryption, Security

MPEG4 Technology and Apps UtilizationMPEG4 Technology and Apps Utilization

Desktop Networked Wired to wireless ubiquity P2PDesktop Networked Wired to wireless ubiquity P2P

PerformancePerformance Resource InequalityResource InequalityPowerPower

Next-gen CODEC’sNext-gen CODEC’s



MRL/TRL

Page 38


MRL/TRLWireless Multimedia:Wireless Multimedia:Robust Video Delivery over Wireless Robust Video Delivery over Wireless Networks (IP, Bluetooth, and beyond)Networks (IP, Bluetooth, and beyond)

Send ACK

CanDecode?

CanDecode?

Packets

PacketizePacketize GenerateFEC

GenerateFEC

ACKGot ACK?Got ACK?

MPEGfile

MPEGfile

WirelessAccess Point


File Selection

Packets

DecodeMPEG

DecodeMPEG

DecodeFEC

DecodeFEC



BufferBuffer

BufferBuffer

TCP

New Results



MRL/TRL

Page 39


MRL/TRLLooking forward…Looking forward… What is the optimal multimedia content distribution What is the optimal multimedia content distribution

mechanism?mechanism? What is the optimal multimedia processing What is the optimal multimedia processing

methodology?methodology? What are the technology building blocks? What are the technology building blocks? Many more questions…Many more questions…

– What are the computing models of the future?What are the computing models of the future?– What are the best algorithms?What are the best algorithms?– What are the What are the performanceperformance criteria? criteria?

Many good problems for researchMany good problems for researchNew framework, new formulation, …New framework, new formulation, …



MRL/TRL

Page 41


MRL/TRLAugmenting Research via University Augmenting Research via University Collaboration & PartnershipsCollaboration & Partnerships

Collaborations (2000-01)Collaborations (2000-01)–UC Berkeley UC Berkeley –Polytech University (NY) Polytech University (NY) –othersothers

PartnershipsPartnerships– InternshipsInternships–Technical exchanges, hosting visitsTechnical exchanges, hosting visits–Courses, resource utilization (e.g. s/w h/w Courses, resource utilization (e.g. s/w h/w

research infrastructure)research infrastructure)



MRL/TRL

Page 42


MRL/TRLSummer Internships @ Intel MRLSummer Internships @ Intel MRL May/June to August/September 2001 (12-13 May/June to August/September 2001 (12-13

weeks, can be renewed to 9 months)weeks, can be renewed to 9 months) Looking for outstanding candidates (MS, Looking for outstanding candidates (MS,

Ph.D., and very exceptional BS students)Ph.D., and very exceptional BS students) Areas:Areas:

1.1. Algorithms research and development (general, Algorithms research and development (general, in various fields) – video, audioin various fields) – video, audio

2.2. Low power researchLow power research

3.3. P2P System researchP2P System research

4.4. Wireless multimedia, communicationsWireless multimedia, communications– Email: Email: [email protected]@intel.com and Matt and Matt

will forward to appropriate researcherswill forward to appropriate researchers

Copyright © 2001 Intel Corporation. On Multimedia Technology & Infrastructure for Emerging...

Documents

Transcript of Copyright © 2001 Intel Corporation. On Multimedia Technology & Infrastructure for Emerging...