Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell...

154
These are confidential sessions—please refrain from streaming, blogging, or taking pictures Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect

Transcript of Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell...

Page 1: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

These are confidential sessions—please refrain from streaming, blogging, or taking pictures

Session 606

Moving to AV Kit and AV Foundation

Sam BushellMedia Frameworks Architect

Page 2: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

Introduction

•A word on QuickTime and QTKit•New AV Kit API• Travel Guide to AV Foundation and AV Kit

Page 3: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

AV Foundation

•We’ve been building a new media infrastructure• Common on iOS and OS X• Focused on modern media formats• Benefits from deep media experience building QuickTime

Page 4: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

Mac OS 7, 8, 9 and Mac OS X1991

QuickTime

Page 5: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

Mac OS X 10.4 TigerQTKit introduced with QuickTime 7.0

QTKit

QuickTime

Page 6: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

Mac OS X 10.6 Snow LeopardOptimized H.264 + AAC: QTMovieOpenForPlaybackAttribute

QTKit

Core MediaQuickTime

Page 7: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

Mac OS X 10.7 LionAV Foundation introduced

AV FoundationQTKit

Core MediaQuickTime

Page 8: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

Core Media

OS X 10.8 Mountain LionVideo Toolbox API introduced

AV FoundationQTKit

Video ToolboxQuickTime

Page 9: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

OS X 10.9 MavericksAV Kit introduced

AV Foundation

AV Kit

QTKit

QuickTime Video Toolbox

Core Media

Page 10: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

OS X 10.9 MavericksAV Kit introduced

AV Foundation

AV Kit

QTKit

QuickTime Video Toolbox

Core Media

Page 11: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

QuickTime and QTKit APIs Are Deprecated

•QuickTime.framework and QTKit.framework APIs deprecated in OS X 10.9•APIs are marked as deprecated in header files• Code will still compile, but with deprecation warnings• Your apps will still run

Page 12: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

QuickTime and QTKit APIs Are Deprecated

•QuickTime.framework and QTKit.framework APIs deprecated in OSX 10.9•APIs are marked as deprecated in header files• Code will still compile, but with deprecation warnings• Your apps will still run

Page 13: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

Still supportedQuickTime Movie Format

•AV Foundation and QuickTime Player still use the QuickTime Movie file format

•Apple is deprecating the QuickTime 7 APIs, not the file format

.MOV

Page 14: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

Still supportedQuickTime Movie Format

•AV Foundation and QuickTime Player still use the QuickTime Movie file format

•Apple is deprecating the QuickTime 7 APIs, not the file format

.MOV

Page 15: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

Still supportedQuickTime Movie Format

•AV Foundation and QuickTime Player still use the QuickTime Movie file format

•Apple is deprecating the QuickTime 7 APIs, not the file format

.MOV .MP4

Page 16: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov
Page 17: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

AV Foundation

Page 18: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

AV Foundation QuickTime

Modern foundation Carbon, QuickDraw, Handles

APIs designed for clients APIs expose implementation

Factored Monolithic

Multithreaded Mostly main-thread-only

Hardware accelerated —

Power efficient —

64-bit native 32-bit only

Page 19: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

AV Foundation QuickTime

Modern foundation Carbon, QuickDraw, Handles

APIs designed for clients APIs expose implementation

Factored Monolithic

Multithreaded Mostly main-thread-only

Hardware accelerated —

Power efficient —

64-bit native 32-bit only

Page 20: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

AV Foundation QuickTime

Modern foundation Carbon, QuickDraw, Handles

APIs designed for clients APIs expose implementation

Factored Monolithic

Multithreaded Mostly main-thread-only

Hardware accelerated —

Power efficient —

64-bit native 32-bit only

Page 21: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

AV Foundation QuickTime

Modern foundation Carbon, QuickDraw, Handles

APIs designed for clients APIs expose implementation

Factored Monolithic

Multithreaded Mostly main-thread-only

Hardware accelerated —

Power efficient —

64-bit native 32-bit only

Page 22: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

AV Foundation QuickTime

Modern foundation Carbon, QuickDraw, Handles

APIs designed for clients APIs expose implementation

Factored Monolithic

Multithreaded Mostly main-thread-only

Hardware accelerated —

Power efficient —

64-bit native 32-bit only

Page 23: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

AV Foundation QuickTime

Modern foundation Carbon, QuickDraw, Handles

APIs designed for clients APIs expose implementation

Factored Monolithic

Multithreaded Mostly main-thread-only

Hardware accelerated —

Power efficient —

64-bit native 32-bit only

Page 24: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

AV Foundation QuickTime

Modern foundation Carbon, QuickDraw, Handles

APIs designed for clients APIs expose implementation

Factored Monolithic

Multithreaded Mostly main-thread-only

Hardware accelerated —

Power efficient —

64-bit native 32-bit only

Page 25: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

AV Foundation QuickTime

Modern foundation Carbon, QuickDraw, Handles

APIs designed for clients APIs expose implementation

Factored Monolithic

Multithreaded Mostly main-thread-only

Hardware accelerated —

Power efficient —

64-bit native 32-bit only

Page 26: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

Supported Media Types

• Video•Audio• Closed captions and subtitles• Chapters• Timecode

Page 27: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

Some examplesNot Supported by AV Foundation

QuickTime VR

RTP Streaming

QT Effects and Filters

Sprite Tracks and Wired Sprites

Flash Tracks

Music (MIDI) Tracks

SMIL

Page 28: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

Supported Codecs

•Delivery codecs■ H.264, AAC, JPEG

•Mezzanine codecs■ Apple ProRes, LPCM

• Camera device codecs■ MPEG-1, MPEG-2, MPEG-4, H.263, DV…

Page 29: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

Not Supported by AV Foundation

Cinepak (“Compact Video”)Animation (“RLE” )Video (“Road Pizza”)Graphics (“SMC”)Sorenson VideoSorenson Video 3Motion JPEG AMotion JPEG BH.261Windows RAWMicrosoft Video 1PixletMACE 3:1MACE 6:1QDesign AudioQDesign Audio 21-bit Indexed-Color RGB2-bit Indexed-Color RGB

4-bit Indexed-Color RGB8-bit Indexed-Color RGB16-bit Direct-Color RGB1-bit Grayscale2-bit Grayscale4-bit GrayscaleSGIMacPaintBMPFLCFlashPixJPEG 2000PDFPhoto CDPNGTGATIFFBlit Codec

Curve RasterizerQuickdraw CodecBlend EffectBlur FilterBrightness and ContrastChannel CompositorChroma Key EffectCloud GeneratorCross Fade EffectEdge Detection FilterEmboss FilterFire GeneratorFilm Noise FilterAlpha Gain FilterGeneral ConvolutionGlass Distortion FilterHSL Balance FilterLens Flare Filter

Gradient Wipe Effect Implode EffectPush EffectRGB Balance FilterRipple FilterSharpen FilterSlide EffectSMPTE Iris EffectSMTPE Radial EffectSMTPE Matrix Wipe EffectWipe EffectColor Style FilterColorSync FilterTravelling Matte EffectExplode EffectZoom Effect

Page 30: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

Your Baby’s First Steps.mov

Sorenson Video 3

QDesign Audio 2

Page 31: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

Your Baby’s First Steps.mov

Sorenson Video 3

QDesign Audio 2

Page 32: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

Your Baby’s First Steps.mov

Sorenson Video 3

QDesign Audio 2

Page 33: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

Your Baby’s First Steps.mov

Sorenson Video 3

QDesign Audio 2

H.264

AAC

Page 34: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

QTMovieModernizer

•Automatically run by QuickTime Player upon discovery of legacy codecs■ Works with third-party QuickTime codec components

•New API in OS X 10.9 so that you can do the same in your apps• Produces a new copy in an AV Foundation-supported format:

■ H.264 + AAC■ Apple ProRes 422 + Linear PCM■ Apple ProRes 4444 + Linear PCM

•Delivered as part of QTKit

Page 35: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

QTKit

QuickTime

Page 36: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

AV Foundation

AV Kit

QTKit

QuickTime Video Toolbox

Core Media

Page 37: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

Introducing AV Kit

Stefan HafenegerMedia Systems Engineer

Page 38: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

AV Kit

AVKit.framework

Page 39: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

AVPlayerView

Page 40: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

DemoAVPlayerView

Page 41: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

AVPlayerView and AV Foundation

AVPlayerView

Page 42: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

AVPlayerView and AV Foundation

AVPlayerView

AVPlayer

Page 43: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

AVPlayerView and AV Foundation

AVPlayerView

AVPlayer

AVPlayerItem

Page 44: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

AVPlayerView and AV Foundation

AVPlayerView

AVPlayer

AVPlayerItem

AVAsset

Page 45: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

Steps to provide content for AVPlayerViewProviding Content

// 1. Create asset from URL.AVAsset *asset = [AVAsset assetWithURL:URL];

// 2. Create player item for asset.AVPlayerItem *playerItem = [AVPlayerItem playerItemWithAsset:asset];

// 3. Create player with player item.AVPlayer *player = [AVPlayer playerWithPlayerItem:playerItem];

// 4. Associate player with player view.[playerView setPlayer:player];

Page 46: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

Steps to provide content for AVPlayerViewProviding Content

// 1. Create asset from URL.AVAsset *asset = [AVAsset assetWithURL:URL];

// 2. Create player item for asset.AVPlayerItem *playerItem = [AVPlayerItem playerItemWithAsset:asset];

// 3. Create player with player item.AVPlayer *player = [AVPlayer playerWithPlayerItem:playerItem];

// 4. Associate player with player view.[playerView setPlayer:player];

Page 47: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

Steps to provide content for AVPlayerViewProviding Content

// 1. Create asset from URL.AVAsset *asset = [AVAsset assetWithURL:URL];

// 2. Create player item for asset.AVPlayerItem *playerItem = [AVPlayerItem playerItemWithAsset:asset];

// 3. Create player with player item.AVPlayer *player = [AVPlayer playerWithPlayerItem:playerItem];

// 4. Associate player with player view.[playerView setPlayer:player];

Page 48: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

Steps to provide content for AVPlayerViewProviding Content

// 1. Create asset from URL.AVAsset *asset = [AVAsset assetWithURL:URL];

// 2. Create player item for asset.AVPlayerItem *playerItem = [AVPlayerItem playerItemWithAsset:asset];

// 3. Create player with player item.AVPlayer *player = [AVPlayer playerWithPlayerItem:playerItem];

// 4. Associate player with player view.[playerView setPlayer:player];

Page 49: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

Steps to provide content for AVPlayerViewProviding Content

// 1. Create asset from URL.AVAsset *asset = [AVAsset assetWithURL:URL];

// 2. Create player item for asset.AVPlayerItem *playerItem = [AVPlayerItem playerItemWithAsset:asset];

// 3. Create player with player item.AVPlayer *player = [AVPlayer playerWithPlayerItem:playerItem];

// 4. Associate player with player view.[playerView setPlayer:player];

Page 50: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

Steps to provide content for AVPlayerViewProviding Content

// 1. Create asset from URL.AVAsset *asset = [AVAsset assetWithURL:URL];

// 2. Create player item for asset.AVPlayerItem *playerItem = [AVPlayerItem playerItemWithAsset:asset];

// 3. Create player with player item.AVPlayer *player = [AVPlayer playerWithPlayerItem:playerItem];

// 4. Associate player with player view.[playerView setPlayer:player];

Page 51: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

One step to provide content for AVPlayerViewProviding Content

// All four steps in one line of code.[playerView setPlayer:[AVPlayer playerWithURL:URL]];

Page 52: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

AVAsset

AVPlayerView and AV Foundation

AVPlayerView

AVPlayer

AVPlayerItem

Page 53: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

AVComposition

AVPlayerView and AV Foundation

AVPlayerView

AVPlayer

AVPlayerItem

Page 54: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

AVPlayerView’s controlsStyle propertyDefining the Look

Page 55: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

AVPlayerView’s controlsStyle propertyDefining the Look

Page 56: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

AVPlayerView’s controlsStyle propertyDefining the Look

Page 57: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

AVPlayerView’s controlsStyle propertyDefining the Look

Page 58: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

AVPlayerView’s controlsStyle propertyDefining the Look

Page 59: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

AVPlayerView’s controlsStyle propertyDefining the Look

Page 60: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

AVPlayerView’s controlsStyle propertyDefining the Look

Page 61: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

AVPlayerView’s controlsStyle propertyDefining the Look

Page 62: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

AVPlayerView’s controlsStyle propertyDefining the Look

Page 63: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

AVPlayerView’s controlsStyle propertyDefining the Look

Page 64: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

AVPlayerView’s controlsStyle propertyDefining the Look

Page 65: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

AVPlayerView’s controlsStyle propertyDefining the Look

Page 66: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

Chapters, languages and subtitlesDynamic Controls

Page 67: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

Chapters, languages and subtitlesDynamic Controls

Page 68: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

Streaming mediaDynamic Controls

Page 69: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

Streaming mediaDynamic Controls

Page 70: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

Sharing service buttonCustomization

Page 71: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

Sharing service buttonCustomization

Page 72: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

Show trim user interfaceTrimming

// Check whether current item can be trimmed.if ([playerView canBeginTrimming]){! // Show trim user interface.! [playerView beginTrimmingWithCompletionHandler:NULL];}

Page 73: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

Show trim user interfaceTrimming

// Check whether current item can be trimmed.if ([playerView canBeginTrimming]){! // Show trim user interface.! [playerView beginTrimmingWithCompletionHandler:NULL];}

Page 74: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

Show trim user interfaceTrimming

// Check whether current item can be trimmed.if ([playerView canBeginTrimming]){! // Show trim user interface.! [playerView beginTrimmingWithCompletionHandler:NULL];}

Page 75: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

Handle trim result after trim user interface is dismissedTrimming

// Check whether current item can be trimmed.if ([playerView canBeginTrimming]){! // Show trim user interface.! [playerView beginTrimmingWithCompletionHandler:^(AVPlayerViewTrimResult result) {! ! // Handle trim result.! ! if (result == AVPlayerViewTrimOKButton)! ! {! ! ! CMTime inPoint = [playerItem reversePlaybackEndTime];! ! ! CMTime outPoint = [playerItem forwardPlaybackEndTime];! ! }! ! else if (result == AVPlayerViewTrimCancelButton)! ! { ... }! }];}

Page 76: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

Handle trim result after trim user interface is dismissedTrimming

// Check whether current item can be trimmed.if ([playerView canBeginTrimming]){! // Show trim user interface.! [playerView beginTrimmingWithCompletionHandler:^(AVPlayerViewTrimResult result) {! ! // Handle trim result.! ! if (result == AVPlayerViewTrimOKButton)! ! {! ! ! CMTime inPoint = [playerItem reversePlaybackEndTime];! ! ! CMTime outPoint = [playerItem forwardPlaybackEndTime];! ! }! ! else if (result == AVPlayerViewTrimCancelButton)! ! { ... }! }];}

Page 77: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

Handle trim result after trim user interface is dismissedTrimming

// Check whether current item can be trimmed.if ([playerView canBeginTrimming]){! // Show trim user interface.! [playerView beginTrimmingWithCompletionHandler:^(AVPlayerViewTrimResult result) {! ! // Handle trim result.! ! if (result == AVPlayerViewTrimOKButton)! ! {! ! ! CMTime inPoint = [playerItem reversePlaybackEndTime];! ! ! CMTime outPoint = [playerItem forwardPlaybackEndTime];! ! }! ! else if (result == AVPlayerViewTrimCancelButton)! ! { ... }! }];}

Page 78: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

Handle trim result after trim user interface is dismissedTrimming

// Check whether current item can be trimmed.if ([playerView canBeginTrimming]){! // Show trim user interface.! [playerView beginTrimmingWithCompletionHandler:^(AVPlayerViewTrimResult result) {! ! // Handle trim result.! ! if (result == AVPlayerViewTrimOKButton)! ! {! ! ! CMTime inPoint = [playerItem reversePlaybackEndTime];! ! ! CMTime outPoint = [playerItem forwardPlaybackEndTime];! ! }! ! else if (result == AVPlayerViewTrimCancelButton)! ! { ... }! }];}

Page 79: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

Handle trim result after trim user interface is dismissedTrimming

// Check whether current item can be trimmed.if ([playerView canBeginTrimming]){! // Show trim user interface.! [playerView beginTrimmingWithCompletionHandler:^(AVPlayerViewTrimResult result) {! ! // Handle trim result.! ! if (result == AVPlayerViewTrimOKButton)! ! {! ! ! CMTime inPoint = [playerItem reversePlaybackEndTime];! ! ! CMTime outPoint = [playerItem forwardPlaybackEndTime];! ! }! ! else if (result == AVPlayerViewTrimCancelButton)! ! { ... }! }];}

Page 80: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

Export trim selectionTrimming

// Get trim in and out points.CMTime inPoint = [playerItem reversePlaybackEndTime];CMTime outPoint = [playerItem forwardPlaybackEndTime];

// Set time range on asset export session.CMTimeRange timeRange = CMTimeRangeFromTimeToTime(inPoint, outPoint);[assetExportSession setTimeRange:timeRange];

Page 81: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

Export trim selectionTrimming

// Get trim in and out points.CMTime inPoint = [playerItem reversePlaybackEndTime];CMTime outPoint = [playerItem forwardPlaybackEndTime];

// Set time range on asset export session.CMTimeRange timeRange = CMTimeRangeFromTimeToTime(inPoint, outPoint);[assetExportSession setTimeRange:timeRange];

Page 82: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

Export trim selectionTrimming

// Get trim in and out points.CMTime inPoint = [playerItem reversePlaybackEndTime];CMTime outPoint = [playerItem forwardPlaybackEndTime];

// Set time range on asset export session.CMTimeRange timeRange = CMTimeRangeFromTimeToTime(inPoint, outPoint);[assetExportSession setTimeRange:timeRange];

Page 83: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

Export trim selectionTrimming

// Get trim in and out points.CMTime inPoint = [playerItem reversePlaybackEndTime];CMTime outPoint = [playerItem forwardPlaybackEndTime];

// Set time range on asset export session.CMTimeRange timeRange = CMTimeRangeFromTimeToTime(inPoint, outPoint);[assetExportSession setTimeRange:timeRange];

Page 84: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

AV Kit Wrap Up

•UI level Cocoa framework for AV Foundation• Standard playback and trim controls through AVPlayerView• Power of AV Foundation without custom view

Page 85: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov
Page 86: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov
Page 87: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

A Travel Guide to AV Foundation and AV Kit

Sam Bushell

Page 88: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

Do You Know the Way to AV Foundation?

•Depends how you use QuickTime■ An easy change for some developers■ Deeper refactoring needed for others

•Not an API-for-API swap• Recent QTKit APIs are similar to their AV Foundation counterparts

Page 89: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

API Areas

•AV Resources• Playback• Exporting media files• Reading and writing media files• Retrieving frames during playback and offline• Editing•Metadata• Capture• Time and media samples• Video compression and decompression

Page 90: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

Basics and Playback

Page 91: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

Representing AV Resources

QuickTime QTKit AV Foundation

MovieTrack / Media

QTMovieQTTrack / QTMedia

AVAssetAVAssetTrack

Page 92: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

Creating Media Objects

QuickTime QTKit AV Foundation

NewMovieFromFileNewMovieFromDataRef

NewMovieFromProperties(etc.)

QTMovie +[movieWithFile:error:]+[movieWithURL:error:]

(etc.)+[AVAsset assetWithURL:]

Page 93: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

Creating Media Objects

QuickTime QTKit AV Foundation

NewMovieFromFileNewMovieFromDataRef

NewMovieFromProperties(etc.)

QTMovie +[movieWithFile:error:]+[movieWithURL:error:]

(etc.)+[AVAsset assetWithURL:]

AVAsset *asset = [AVAsset assetWithURL:...];[asset loadValuesAsynchronouslyForKeys:@[@”tracks”,@”duration”] ! ! ! completionHandler:^{...}];

Page 94: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

Playback

QuickTime QTKit AV Foundation

SetMovieRateSetMovieTime

QTMovie - play- stop- setCurrentTime:- stepForward- stepBackward

AVPlayer / AVQueuePlayer- play- pauserate property

AVPlayerItem- seekToTime: and variants- stepByCount:

Page 95: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

Playback

QuickTime QTKit AV Foundation

SetMovieRateSetMovieTime

QTMovie - play- stop- setCurrentTime:- stepForward- stepBackward

AVPlayer / AVQueuePlayer- play- pauserate property

AVPlayerItem- seekToTime: and variants- stepByCount:

[item seekToTime:t toleranceBefore:b toleranceAfter:a completionHandler:^{...}];

Page 96: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

Playback in AppKit NSViews

AV KitQTKit AV Kit

QTMovieView- play:- pause:

AVPlayerViewcontrolsStyle propertyplayer property

AVPlayer- play- pauserate property

Page 97: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

Playback via Core Animation

QTKit AV Foundation

QTMovieLayer- setMovie:

AVPlayerLayerplayer property

AVSynchronizedLayerplayerItem property

Page 98: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

Playback via Core Animation

QTKit AV Foundation

QTMovieLayer- setMovie:

AVPlayerLayerplayer property

AVSynchronizedLayerplayerItem property

Preparing and Presenting Media for Accessibility Nob HillWednesday 10:15AM

Page 99: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

Authoring and Editing

Page 100: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

Media Export

QuickTime ConvertMovieToDataRefMovieExportToDataRef

QTKitQTMovie

- writeToFile:withAttributes:

QTExportSession- initWithMovie:exportOptions:outputURL:error:

AV FoundationAVAssetExportSession

- initWithAsset:presetName:

AVAssetReader and AVAssetWriter

Page 101: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

Reading Media Files

QuickTime AV Foundation

GetMediaSample2AVAssetReader

AVAssetReaderOutput (per-track)(created with outputSettings:nil)

SetMovieGWorld / SetMovieVisualContext SetMovieTimeMoviesTask

MovieAudioExtraction

AVAssetReaderAVAssetReaderOutput (per-track)(non-nil outputSettings)

Page 102: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

AVAssetReaderDeeply multithreaded

AVAssetReader

AAC

H.264

PCM

YCbCr

AACAAC PCM

H.264H.264 YUVYUV

PCM

File I/O

File I/O

Decoder

Decoder

Page 103: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

Writing Media Files

QuickTime AddMediaSample2 ... AddMovieToStorageMovieExportFromProceduresToDataRef

QTKitQTMovie

- initToWritableDataReference:error:- addImage:forDuration:withAttributes:

AV FoundationAVAssetWriter

AVAssetWriterInput (per-track)

AVOutputSettingsAssistant

Page 104: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

Writing Media Files

QuickTime AddMediaSample2 ... AddMovieToStorageMovieExportFromProceduresToDataRef

QTKitQTMovie

- initToWritableDataReference:error:- addImage:forDuration:withAttributes:

AV FoundationAVAssetWriter

AVAssetWriterInput (per-track)

AVOutputSettingsAssistant

Page 105: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

Writing Media Files

QuickTime AddMediaSample2 ... AddMovieToStorageMovieExportFromProceduresToDataRef

QTKitQTMovie

- initToWritableDataReference:error:- addImage:forDuration:withAttributes:

AV FoundationAVAssetWriter

AVAssetWriterInput (per-track)

AVOutputSettingsAssistant

Page 106: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

Preparing and Presenting Media for Accessibility Nob HillWednesday 10:15AM

Writing Media Files

QuickTime AddMediaSample2 ... AddMovieToStorageMovieExportFromProceduresToDataRef

QTKitQTMovie

- initToWritableDataReference:error:- addImage:forDuration:withAttributes:

AV FoundationAVAssetWriter

AVAssetWriterInput (per-track)

AVOutputSettingsAssistant

Page 107: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

AVAssetWriter

AVAssetWriterDeeply multithreaded

AACAACAAC

H.264H.264H.264YCbCrYUVYUV

PCMPCMPCM

Encoder

Encoder

Page 108: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

AVAssetWriter

AVAssetWriterDeeply multithreaded

AACAACAAC

H.264H.264H.264YCbCrYUVYUV

PCMPCMPCM

Encoder

Encoder

Page 109: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

AVAssetWriter

AVAssetWriterDeeply multithreaded

AACAACAACPCMPCMPCM

Encoder

Encoder

Page 110: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

AVAssetWriter

AVAssetWriterDeeply multithreaded

AACAACAACPCMPCMPCMSTOP

GO

Encoder

Encoder

Page 111: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

Getting Video Frames During Playback

QuickTime SetMovieVisualContext

QTKit QTMovie - setVisualContext:

AV Foundation AVPlayerItemVideoOutput

Page 112: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

Accessing Audio During Playback

QuickTime QTAudioContextRegisterInsertSetMovieAudioContext

QTKit —

AV Foundation MTAudioProcessingTapAVAudioMix.audioTapProcessor property

Page 113: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

Getting Still Images from Video

QTKit QTMovie - frameImageAtTime:

AV FoundationAVAssetImageGenerator

- copyCGImageAtTime:actualTime:error:- generateCGImagesAsynchronouslyForTimes: completionHandler:

Page 114: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

Editing

QuickTimeInsertMovieSegmentDeleteMovieSegmentScaleMovieSegment

InsertTrackSegmentDeleteTrackSegmentScaleTrackSegment

QTKitQTMovie

- insertSegmentOfMovie:...- deleteSegment:- scaleSegment:newDuration:

QTTrack- insertSegmentOfTrack:...- deleteSegment:- scaleSegment:newDuration:

AV FoundationAVMutableComposition

- insertTimeRange:ofAsset:...- removeTimeRange:- scaleTimeRange:toDuration:

AVMutableCompositionTrack- insertTimeRange:ofTrack:...- removeTimeRange:- scaleTimeRange:toDuration:

Page 115: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

Source Movies

Edit List

EditingEdit lists and sample tables

Play from time 10 for 5 seconds

Play from time 0 for 2 seconds

Play from time 3 for 4 seconds

Page 116: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

Source Movies

Edit List

Editing

Sample TableByte offset and size for each sample

Byte offset and size for each sample

Byte offset and size for each sample

Edit lists and sample tables

Play from time 10 for 5 seconds

Play from time 0 for 2 seconds

Play from time 3 for 4 seconds

Required by QuickTime API

Page 117: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

Not required byAV Foundation

Source Movies

Edit List

EditingEdit lists and sample tables

Play from time 10 for 5 seconds

Play from time 0 for 2 seconds

Play from time 3 for 4 seconds

Page 118: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

Metadata

QTKit AV Foundation

QTMovie- commonMetadata- availableMetadataFormats- metadataForFormat:

QTMetadataItemproperties: key, keySpace, locale, time, value, extraAttributes (etc.)

AVAssetcommonMetadata propertyavailableMetadataFormats property- metadataForFormat:

AVMetadataItemproperties: key, keySpace, locale, time, value, extraAttributes (etc.)

Page 119: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

Metadata

[asset loadValuesAsynchronouslyForKeys:@[@”commonMetadata”] ! ! ! completionHandler:^{...}];

QTKit AV Foundation

QTMovie- commonMetadata- availableMetadataFormats- metadataForFormat:

QTMetadataItemproperties: key, keySpace, locale, time, value, extraAttributes (etc.)

AVAssetcommonMetadata propertyavailableMetadataFormats property- metadataForFormat:

AVMetadataItemproperties: key, keySpace, locale, time, value, extraAttributes (etc.)

Page 120: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

Capture

Page 121: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

Capture

QuickTime QTKit AV Foundation

Sequence GrabberQTCaptureSessionQTCaptureInputQTCaptureOutput

QTCaptureConnection

AVCaptureSessionAVCaptureInputAVCaptureOutput

AVCaptureConnection

Page 122: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

Capture ConnectionsBetween inputs and outputs

Page 123: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

Capture ConnectionsBetween inputs and outputs

Input

Page 124: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

Capture ConnectionsBetween inputs and outputs

Input

Preview

Page 125: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

Capture ConnectionsBetween inputs and outputs

Output

Input

Preview

Page 126: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

Capture ConnectionsBetween inputs and outputs

Output

Video connection

Input

Preview

Page 127: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

Capture ConnectionsBetween inputs and outputs Audio connection

Output

Video connection

Input

Preview

Page 128: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

Capture ConnectionsBetween inputs and outputs Audio connection

Output

Video connection

Input

Preview

Page 129: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

Capture ConnectionsBetween inputs and outputs Audio connection

Output

Video connection

Input

Preview

Page 130: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

Capture ConnectionsBetween inputs and outputs Audio connection

Output

Video connection

Input

Preview

Page 131: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

Capture ConnectionsBetween inputs and outputs Audio connection

Output

Video connection

Input

Preview

Page 132: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

Capture ConnectionsBetween inputs and outputs Audio connection

Output

Video connection

Input

Preview

Page 133: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

Input/Output

QTKit AV Foundation

QTCaptureDeviceInput AVCaptureDeviceInputAVCaptureScreenInput

QTCaptureAudioPreviewOutputQTCaptureVideoPreviewOutput

QTCaptureDecompressedAudioOutputQTCaptureDecompressedVideoOutput

QTCaptureMovieFileOutput

AVCaptureAudioPreviewOutput

AVCaptureAudioDataOutputAVCaptureVideoDataOutputAVCaptureMovieFileOutputAVCaptureAudioFileOutputAVCaptureStillImageOutput

Page 134: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

Input/Output

QTKit AV Foundation

QTCaptureDeviceInput AVCaptureDeviceInputAVCaptureScreenInput

QTCaptureAudioPreviewOutputQTCaptureVideoPreviewOutput

QTCaptureDecompressedAudioOutputQTCaptureDecompressedVideoOutput

QTCaptureMovieFileOutput

AVCaptureAudioPreviewOutput

AVCaptureAudioDataOutputAVCaptureVideoDataOutputAVCaptureMovieFileOutputAVCaptureAudioFileOutputAVCaptureStillImageOutput

Page 135: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

Input/Output

QTKit AV Foundation

QTCaptureDeviceInput AVCaptureDeviceInputAVCaptureScreenInput

QTCaptureAudioPreviewOutputQTCaptureVideoPreviewOutput

QTCaptureDecompressedAudioOutputQTCaptureDecompressedVideoOutput

QTCaptureMovieFileOutput

AVCaptureAudioPreviewOutput

AVCaptureAudioDataOutputAVCaptureVideoDataOutputAVCaptureMovieFileOutputAVCaptureAudioFileOutputAVCaptureStillImageOutput

Page 136: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

Input/Output

QTKit AV Foundation

QTCaptureDeviceInput AVCaptureDeviceInputAVCaptureScreenInput

QTCaptureAudioPreviewOutputQTCaptureVideoPreviewOutput

QTCaptureDecompressedAudioOutputQTCaptureDecompressedVideoOutput

QTCaptureMovieFileOutput

AVCaptureAudioPreviewOutput

AVCaptureAudioDataOutputAVCaptureVideoDataOutputAVCaptureMovieFileOutputAVCaptureAudioFileOutputAVCaptureStillImageOutput

Page 137: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

Input/Output

QTKit AV Foundation

QTCaptureDeviceInput AVCaptureDeviceInputAVCaptureScreenInput

QTCaptureAudioPreviewOutputQTCaptureVideoPreviewOutput

QTCaptureDecompressedAudioOutputQTCaptureDecompressedVideoOutput

QTCaptureMovieFileOutput

AVCaptureAudioPreviewOutput

AVCaptureAudioDataOutputAVCaptureVideoDataOutputAVCaptureMovieFileOutputAVCaptureAudioFileOutputAVCaptureStillImageOutput

Page 138: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

Input/Output

QTKit AV Foundation

QTCaptureDeviceInput AVCaptureDeviceInputAVCaptureScreenInput

QTCaptureAudioPreviewOutputQTCaptureVideoPreviewOutput

QTCaptureDecompressedAudioOutputQTCaptureDecompressedVideoOutput

QTCaptureMovieFileOutput

AVCaptureAudioPreviewOutput

AVCaptureAudioDataOutputAVCaptureVideoDataOutputAVCaptureMovieFileOutputAVCaptureAudioFileOutputAVCaptureStillImageOutput

Page 139: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

Device Access

QTKit AV Foundation

QTCaptureDevice+ inputDevices+ inputDevicesWithMediaType:+ defaultInputDeviceWithMediaType:+ deviceWithUniqueID:

AVCaptureDevice+ devices+ devicesWithMediaType:+ defaultDeviceWithMediaType:+ deviceWithUniqueID:

Page 140: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

Display of What’s Being Recorded

QTKit AV Foundation

QTCaptureLayerQTCaptureView

AVCaptureVideoPreviewLayer

Page 141: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

Low-Level Media Objects

Page 142: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

Representation of Time Values

64-bit time value (numerator)

32-bit time scale (denominator)

Page 143: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

Representation of Time Values

64-bit time value (numerator)

32-bit time scale (denominator)

QuickTime QTKit AV Foundationand Core Media

TimeRecord (64-bit)TimeValue (32-bit)

QTTimeQTTimeRange

CMTimeCMTimeRange

CMTimeMapping

Page 144: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

Representation of Moving Time

QuickTime AV Foundationand Core Media

ClockTimeBase

CMClockCMTimebase

Page 145: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

QuickTime QTKit AV Foundationand Core Media

SampleDescriptionHandle

ImageDescriptionHandleSoundDescriptionHandle

(etc.)

QTFormatDescription CMFormatDescription

CMVideoFormatDescriptionCMAudioFormatDescription

(etc.)

QTSampleBuffer CMSampleBuffer

Describing Compressed Media Samples

Page 146: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

Video Compression and Decompression

QuickTime’s Image Compression Manager

AV Foundation’s Video Toolbox

CompressSequenceBeginDecompressSequenceBegin

ICMCompressionSessionICMDecompressionSession

VTCompressionSessionVTDecompressionSessionVTPixelTransferSession

Page 147: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

Still Image I/O

QuickTime ImageIO

GraphicsImporterGraphicsExporter

CGImageSourceCGImageDestination

Page 148: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

AVAsset, AVAssetTrack

AVPlayer, AVPlayerItem, AVPlayerItemTrack

AVQueuePlayer, AVPlayerLayer, AVPlayerView

AVSynchronizedLayer

AVAssetExportSession

AVAssetReader, AVAssetReaderInput

AVAssetWriter, AVAssetWriterOutput

AVOutputSettingsAssistant

AVPlayerItemVideoOutput

AVAudioMix, MTAudioProcessingTap

AVAssetImageGenerator

AVComposition, AVCompositionTrack

AVMetadataItem

AVCaptureSession, AVCaptureConnection

AVCaptureDeviceInput, AVCaptureScreenInput

AVCaptureAudioPreviewOutput, AVCaptureAudioDataOutput, AVCaptureVideoDataOutput, AVCaptureMovieFileOutput, AVCaptureAudioFileOutput, AVCaptureStillImageOutput

AVCaptureVideoPreviewLayer

CMTime, CMTimeRange, CMTimeMapping

CMClock, CMTimebase

CMFormatDescription, CMVideoFormatDescription, CMAudioFormatDescription

CMSampleBuffer, CMBlockBuffer

VTCompressionSession, VTDecompressionSession, VTPixelTransferSession

Page 149: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

Summary

•QuickTime.framework and QTKit.framework APIs deprecated in OS X 10.9■ Reminding you to make the transition■ Your apps will still run

•AV Foundation and AV Kit are our modern media frameworks■ And they’re better!

•QuickTime Movie file format is still our primary file format•QTMovieModernizer•AVPlayerView

Page 150: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

We welcome your feedback

Page 151: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

More Information

John GeleynseDirector, Technology [email protected]

DocumentationAV Foundation Programming Guidehttps://developer.apple.com/library/mac/#documentation/AudioVideo/Conceptual/AVFoundationPG/

Technical Note TN2300: Transitioning QTKit Code to AV Foundationhttps://developer.apple.com/library/mac/technotes/tn2300/

Apple Developer Forumshttp://devforums.apple.com

Page 152: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

What's New in Camera Capture Nob HillWednesday 11:30AM

Preparing and Presenting Media for Accessibility Nob HillWednesday 10:15AM

Related Sessions

Advanced Editing with AV Foundation MarinaThursday 9:00AM

Page 153: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov

AV Foundation Lab Media Lab BFriday 9:00AM

AV Foundation Lab Media Lab BThursday 2:00PM

OS X and iOS Capture Lab Media Lab BThursday 9:00AM

Labs

Page 154: Moving to AV Kit and AV Foundation · Session 606 Moving to AV Kit and AV Foundation Sam Bushell Media Frameworks Architect. Introduction ... Zoom Effect. Your Baby’s First Steps.mov