Media extensions

Media extensionsMarcin Stankiewicz and Matthew HowardDevelopment - Multimedia Platform TeamRichard FricksWindows Scenario Adoption TeamSession 4-104

Engaging with rich media—whether watching a movie, video chatting, or playing music—is one of the most prevalent and enjoyable things we do on our PCs today.

Drag picture to placeholder or click icon to add

Architecture overviewMedia Foundation pipelineMedia extensionsVisual Studio 2012Media source demoStreaming media to JavaScript

Agenda

Media Application Architecture

Media Foundation

Media enginePlayback and preview

Windows Store app

Capture engineCapture

Sharing enginePlayTo

Transcode

Windows Runtime (WinRT)

C#MediaElement

JavaScript – HTML5Video tag, audio tag

Media namespace• Capture• Transcode• PlayTo• PlaybackManager• MediaExtensionMana

ger

Media source

Video decoder

Video effect

Video encoder Video sink

Audio decoder

Audio effect

Audio encoder Audio sink

Video

Audio

Media Foundation pipeline

Media app architecture

Media enginePlayback and preview

Capture engineCapture

Sharing enginePlay To

Transcode

Media Foundation

Media source

Decoder Effect SinkEncoder

Media app architecture

Media source

Video decoder

Video effect


Audio decoder

Audio effect


Video

Audio

Media Foundation pipelineMedia source

Decoder Effect SinkEncoder

http://msdn.microsoft.com/en-us/library/windows/apps/hh452756.aspx

• Can be either native C++ DLL or a native C++/CX DLL• The DLL must ship in your app package• You can call WinRT APIs but you can use ONLY those desktop APIs that are

marked for Windows Store apps:



Registration processMedia source

Video decoder

Video effect


Audio decoder

Audio effect


Video

Audio


Windows.Media.MediaExtensionManager

.RegisterAudioDecoder()

.RegisterVideoDecoder()

.RegisterAudioEncoder()

.RegisterVideoEncoder()

.RegisterSchemeHandler()

.RegisterByteStreamHandler()?

Media source

Video decoder

Video effect


Audio decoder

Audio effect


Video

Audio


Effects are registered using the class your code uses to process the media:MediaElement AddAudioEffect() VideoTag msInsertAudioEffect()

AddVideoEffect() msInsertVideoEffect()

MediaTranscoder AddAudioEffect() AudioTag msInsertAudioEffect()

AddVideoEffect()

MediaCapture AddEffectAsync()

Registration process: Effects

Media source

Video decoder

Video effect


Audio decoder

Audio effect


Video

Audio


Sinks are supported only by MediaCapture:MediaCapture StartRecordToCustomSinkAsync()

StartPreviewToCustomSinkAsync()

Registration process: Sinks

Media source

Video decoder

Video effect


Audio decoder

Audio effect


Video

Audio


Windows.Media.MediaExtensionManager






.RegisterByteStreamHandler()?






.RegisterByteStreamHandler()

Registration Process: Media source

ByteStreamHandler• New container format (WebM, MKV)

SchemeHandler• Generating your own streams• Media stored in a location not

accessible to inbox ByteStream• Custom streaming protocol

Which to use?

IMFByteStreamHandlerInterface

Method DescriptionBeginCreateObject Begins an asynchronous request to create a media source from

a byte stream.CancelObjectCreation Cancels the current request to create a media source.

EndCreateObjectCompletes an asynchronous request to create an object from a URL. Completes an asynchronous request to create a media source.

GetMaxNumberOfBytesRequiredForResolution

Retrieves the maximum number of bytes needed to create the media source or determine that the byte stream handler cannot parse this stream.

HRESULT BeginCreateObject( [in] IMFByteStream *pByteStream,[in] LPCWSTR pwszURL,[in] DWORD dwFlags,[in] IPropertyStore *pProps,[out] IUnknown **ppIUnknownCancelCookie,[in] IMFAsyncCallback *pCallback,[in] IUnknown *punkState );


MediaElement.Source = new Uri(“video.mv4");

HRESULT EndCreateObject( [in] IMFAsyncResult *pResult,[out] MF_OBJECT_TYPE *pObjectType,[out] IUnknown **ppObject );

typedef enum MF_OBJECT_TYPE { MF_OBJECT_MEDIASOURCE,MF_OBJECT_BYTESTREAM,MF_OBJECT_INVALID } MF_OBJECT_TYPE;


SchemeHandlers and Byte-StreamHandlers

Byte-StreamHandler

Media source

Video decoder

Video effect


Audio decoder

Audio effect


Video

Audio


IMFSchemeHandlerInterface

Method DescriptionBeginCreateObject Begins an asynchronous request to create an object from a

URL.CancelObjectCreation Cancels the current request to create an object from a URL.

EndCreateObject Completes an asynchronous request to create an object from a URL.

HRESULT BeginCreateObject( [in] LPCWSTR pwszURL,[in] DWORD dwFlags,[in] IPropertyStore *pProps,[out] IUnknown

**ppIUnknownCancelCookie,[in] IMFAsyncCallback *pCallback,[in] IUnknown *punkState );


MediaElement.Source = new Uri("myscheme://square");

HRESULT EndCreateObject( [in] IMFAsyncResult *pResult,[out] MF_OBJECT_TYPE *pObjectType,[out] IUnknown **ppObject );


typedef enum MF_OBJECT_TYPE { MF_OBJECT_MEDIASOURCE,MF_OBJECT_BYTESTREAM,MF_OBJECT_INVALID } MF_OBJECT_TYPE;

SchemeHandlers and Byte-StreamHandlers

Byte-StreamHandler

Media source

Video decoder

Video effect


Audio decoder

Audio effect


Video

Audio


SchemeHandlers and Byte-StreamHandlersSchemeHandler

Byte-StreamHandler

Media source

Video decoder

Video effect


Audio decoder

Audio effect


Video

Audio


ByteStream

What is a ByteStream?

Represents a byte stream from some data source:• Local file• Network file• Some other source

This interface supports typical stream operations such as reading, writing, and seeking.

IMFByteStream Interface

Method GetCurrentPositionBeginRead GetLengthBeginWrite IsEndOfStreamClose ReadEndRead SeekEndWrite SetCurrentPositionFlush SetLengthGetCapabilities Write


Byte-StreamHandler

Media source

Video decoder

Video effect


Audio decoder

Audio effect


Video

Audio


ByteStream


Byte-StreamHandler

Media source

Video decoder

Video effect


Audio decoder

Audio effect


Video

Audio


ByteStream

SourceResolver


Media source

Video decoder

Video effect


Audio decoder

Audio effect


Video

Audio

SchemeHandler

Byte-StreamHandlerByteStream

SourceResolver

Marcin Stankiewicz Development - Multimedia Platform Team

Demo: Building a media extension• How to build an extension using C++/CX• The proper use of WRL to implement a

SchemeHandler• Instantiating a media source from a SchemeHandler • Using the PropertySet class to implement two-way

communication between the app and the extension• PropertySet change notification to support dynamic

updates• Direct3D rendering

Recap of Demo• How to build an extension using C++/CX• The proper use of WRL to implement a

SchemeHandler• Instantiating a media source from a

SchemeHandler • Using the PropertySet class to implement two-

way communication between the app and the extension

• PropertySet change notification to support dynamic updates

• Direct3D rendering


Media source

Video decoder

Video effect


Audio decoder

Audio effect


Video

Audio

SchemeHandler

Byte-StreamHandlerByteStream

SourceResolver

Matthew HowardDevelopment - Multimedia Platform Team

Demo: Streaming media to an app• How to implement a sink media extension for

audio• Capturing uncompressed audio• Processing audio buffers in a JavaScript app• Writing audio to a wave file• Asynchronous processing techniques

Recap of Demo• How to implement a sink media extension for

audio• Capturing uncompressed audio• Processing audio buffers in a JavaScript app• Writing audio to a wave file• Asynchronous processing techniques

Need more information?

Come to our question-and-answer session!3-106 Chalk Talk: Media appsThursday 4:15pm, B33 Hood

Media-related sessions3-117 10/31/2012 17:15300 - Advanced B33 Hood Key technologies for building advanced media apps3-122 11/1/2012 14:30 300 - Advanced B33 Hood Media app case studies3-106 11/1/2012 16:15 300 - Advanced B33 Hood Chalk Talk: Media apps

Resources

Windows 8 media overviewhttp://blogs.msdn.com/b/b8/archive/2012/06/08/building-a-rich-and-extensible-media-platform.aspx

Media extensions SDK samplehttp://code.msdn.microsoft.com/windowsapps/Media-extensions-sample-7b466096

Windows 8 multimedia desktop APIs available to Windows Store appshttp://msdn.microsoft.com/en-us/library/windows/apps/hh452756.aspx

SchemeHandlers and ByteStreamHandlershttp://msdn.microsoft.com/en-us/library/windows/desktop/aa371872(v=vs.85).aspx

Writing a Custom Media Sourcehttp://msdn.microsoft.com/en-us/library/windows/desktop/ms700134(v=vs.85).aspx

http://blogs.msdn.com/b/b8/archive/2012/06/08/building-a-rich-and-extensible-media-platform.aspx



http://code.msdn.microsoft.com/windowsapps/Media-extensions-sample-7b466096

http://code.msdn.microsoft.com/windowsapps/Media-extensions-sample-7b466096




http://msdn.microsoft.com/en-us/library/windows/desktop/aa371872(v=vs.85).aspx



http://msdn.microsoft.com/en-us/library/windows/desktop/ms700134(v=vs.85).aspx



• Develop: http://msdn.microsoft.com/en-US/windows/apps/br229512

• Design: http://design.windows.com/

• Samples: http://code.msdn.microsoft.com/windowsapps/Windows-8-Modern-Style-App-Samples

• Videos: http://channel9.msdn.com/Windows

Resources

Please submit session evals by using the Build Windows 8 appor at http://aka.ms/BuildSessions

http://msdn.microsoft.com/en-US/windows/apps/br229512



http://design.windows.com/

http://code.msdn.microsoft.com/windowsapps/Windows-8-Modern-Style-App-Samples




http://channel9.msdn.com/Windows

http://channel9.msdn.com/Windows

http://aka.ms/BuildSessions

© 2012 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Media extensions

Documents

Transcript of Media extensions