G6DPMM - Lecture 12
description
Transcript of G6DPMM - Lecture 12
School of Computer Science & Information Technology
G6DPMM - Lecture 12G6DPMM - Lecture 12
Digital VideoDigital Video
Sourcing Digital Video Film as DigitalFilm as Digital
Becoming more common as costs fallBecoming more common as costs fall May be compressed in the camera or uncompressedMay be compressed in the camera or uncompressed Stored on magnetic tapeStored on magnetic tape
D1, D2, D3, D5, D9 (Digital-S)D1, D2, D3, D5, D9 (Digital-S) Digital Betacam, BetacamSX, BetacamIMXDigital Betacam, BetacamSX, BetacamIMX DVCAM DVCPRODVCAM DVCPRO DV / MiniDV (consumer)DV / MiniDV (consumer) Digital-8 (uses Hi-8 tapes)Digital-8 (uses Hi-8 tapes)
Compressed vs UncompressedCompressed vs Uncompressed Film as AnalogueFilm as Analogue
Digitise with capture board (frame grabber)Digitise with capture board (frame grabber) Wildly variable in price and qualityWildly variable in price and quality
Digital Video Playback Playback Software / Multimedia FrameworkPlayback Software / Multimedia Framework Windows Media Player (Microsoft)Windows Media Player (Microsoft)
Bundled with Windows (3.1 - XP)Bundled with Windows (3.1 - XP) Free versions for Mac OSX and SolarisFree versions for Mac OSX and Solaris Media framework – includes video infrastructureMedia framework – includes video infrastructure File formats (e.g. AVI / WMV)File formats (e.g. AVI / WMV)
QuicktimeQuicktime Bundled with Macintosh (OS 7 – X)Bundled with Macintosh (OS 7 – X) Free version for WindowsFree version for Windows Media framework – includes video infrastructureMedia framework – includes video infrastructure MOV file formatMOV file format
RealNetworksRealNetworks RealPlayer (Windows / Macintosh / Linux / SolarisRealPlayer (Windows / Macintosh / Linux / Solaris Originally a streaming technology, now a full blown media Originally a streaming technology, now a full blown media
frameworkframework
Digital Video Compression Video consists of a time-series of framesVideo consists of a time-series of frames Frames are still imagesFrames are still images
NB still image compression techniquesNB still image compression techniques Sequences of frames may be compressed by:Sequences of frames may be compressed by:
Motion estimationMotion estimation Only updating the parts of the frame that have moved Only updating the parts of the frame that have moved E.g. consider a talking head!E.g. consider a talking head!
Motion compensationMotion compensation Finding static parts of the frame and moving them aroundFinding static parts of the frame and moving them around E.g. consider a pan or zoomE.g. consider a pan or zoom
Compression Codecs Playback software (media framework) does not Playback software (media framework) does not
usually contain compression/decompression usually contain compression/decompression softwaresoftware
Compression software implementing a specific Compression software implementing a specific algorithm is called a algorithm is called a codeccodec
Codecs are required by the framework – many Codecs are required by the framework – many different codecs can be installed into a given different codecs can be installed into a given frameworkframework
NB compatibility and distribution (legality) NB compatibility and distribution (legality) issuesissues
Common Codecs IndeoIndeo CinepakCinepak P*64P*64 SorensonSorenson ClearVideoClearVideo VDOwaveVDOwave RealVideoRealVideo MPEGMPEG
NB this is NB this is notnot a single codec a single codec MPEG also has many spin-offs – such as Microsoft MPEG also has many spin-offs – such as Microsoft
WMV or DivXWMV or DivX
Video Compression Uncompressed video is expensiveUncompressed video is expensive
Hard to store – very large volumes of dataHard to store – very large volumes of data Expensive equipment required to manipulateExpensive equipment required to manipulate
Compression is Compression is alwaysalways lossy lossy Multi-way trade-offMulti-way trade-off
File size / transfer rate (bandwidth)File size / transfer rate (bandwidth) Frame rate (typically 15fps for multimedia)Frame rate (typically 15fps for multimedia) Image qualityImage quality Frame sizeFrame size
Different codecs have differing strengths and Different codecs have differing strengths and weaknesses (e.g. Indeo vs Cinepak)weaknesses (e.g. Indeo vs Cinepak)
Indeo Algorithms developed for DVI – Intel i750 Algorithms developed for DVI – Intel i750
chipsetchipset Indeo – implemented in softwareIndeo – implemented in software Proprietary intel algorithmsProprietary intel algorithms Two levels of compressionTwo levels of compression
PLV (Production Level Video) – broadcast qualityPLV (Production Level Video) – broadcast quality RTV (Real Time Video)RTV (Real Time Video)
Typical compression ratio of 160:1Typical compression ratio of 160:1 Beware versions (5.1 safe on modern machines, Beware versions (5.1 safe on modern machines,
3.2 fairly ubiquitous)3.2 fairly ubiquitous)
MPEG Moving Picture Expert GroupMoving Picture Expert Group Established by ISO in 1988Established by ISO in 1988 Remit – define standards for movie with soundtracksRemit – define standards for movie with soundtracks
Digital televisionDigital television Video phonesVideo phones Consumer electronicsConsumer electronics Digital cinemaDigital cinema Video gamesVideo games
Family of standardsFamily of standards Always lossyAlways lossy Discrete Cosine Transform (DCT) algorithmDiscrete Cosine Transform (DCT) algorithm
MPEG-1 (1993) The original MPEG standardThe original MPEG standard Objective: deliver video and audio from CD-Objective: deliver video and audio from CD-
ROM with comparable quality to VHS tape and ROM with comparable quality to VHS tape and CD-AudioCD-Audio
Encodes an audiovisual signal in about 1.5 MbpsEncodes an audiovisual signal in about 1.5 Mbps Images are progressively scannedImages are progressively scanned
This fundamentally limits qualityThis fundamentally limits quality Motion compensationMotion compensation
I-Frames (reference)I-Frames (reference) P-Frames (predictive)P-Frames (predictive) B-Frames (bidirectional predictive)B-Frames (bidirectional predictive)
MPEG Frame Types I-FramesI-Frames
Reference frameReference frame Complete image – no missing informationComplete image – no missing information Subsequent P-frames are based upon an I-frameSubsequent P-frames are based upon an I-frame
P-FramesP-Frames Only contain the difference between themselves and the Only contain the difference between themselves and the
previous P-frame or I-frameprevious P-frame or I-frame Anything that does not change is not storedAnything that does not change is not stored
B-FramesB-Frames Contain the difference between themselves and both frames on Contain the difference between themselves and both frames on
either sideeither side B-frames are not always implemented because of memory B-frames are not always implemented because of memory
requirementsrequirements
The MPEG GOP GOP StructureGOP Structure Group of PicturesGroup of Pictures Specifies I / P frame structure (e.g. 1 I frame per 11 P Specifies I / P frame structure (e.g. 1 I frame per 11 P
frames.frames.
IPPPPPPPPPPPIPPPPPPPPPPPIPPPPPPPPPPPIIPPPPPPPPPPPIPPPPPPPPPPPIPPPPPPPPPPPI
The fewer I frames the better the compression…The fewer I frames the better the compression… but frame to frame calculations are less accurate – leading to but frame to frame calculations are less accurate – leading to
artefactsartefacts PEL (picture elements)PEL (picture elements)
Block of pixels, used as a basis for compressionBlock of pixels, used as a basis for compression Motion of PEL is predicted based on I, P or B framesMotion of PEL is predicted based on I, P or B frames
The MPEG-1 standard Five parts (separate ISO standards)Five parts (separate ISO standards)
1. Systems1. Systems
2. Video2. Video
3. Audio (NB MPEG 1, level 3 – MP3)3. Audio (NB MPEG 1, level 3 – MP3)
4. Conformance testing4. Conformance testing
5. C implementation (encoding and decoding)5. C implementation (encoding and decoding)
MPEG Audio MP3 is MPEG 1, audio layer 3MP3 is MPEG 1, audio layer 3
Layer 1: 384, 4:1 compressionLayer 1: 384, 4:1 compression Layer 2: 192 – 256kbit/s, 6:1..8:1 compressionLayer 2: 192 – 256kbit/s, 6:1..8:1 compression Layer 3: 112 – 128 kbit/s, 10:1..12:1 compressionLayer 3: 112 – 128 kbit/s, 10:1..12:1 compression
CompressionCompression Modified discrete cosine transfom (MDCT)Modified discrete cosine transfom (MDCT) 32 band polyphase quadrature filter (PQF)32 band polyphase quadrature filter (PQF) Alias reduction postprocessingAlias reduction postprocessing
Severe design limitations of MP3Severe design limitations of MP3 Bitrate, time resolution, lack of scaleability – Bitrate, time resolution, lack of scaleability –
especially for high frequenciesespecially for high frequencies
Applications of MPEG-1 MPEG-1 compression is cheap and rapidMPEG-1 compression is cheap and rapid Consumer DevicesConsumer Devices Multimedia / CD-ROMMultimedia / CD-ROM Internet VideoInternet Video Video CD (production is extremely cheap)Video CD (production is extremely cheap) Internet audio (MP2)Internet audio (MP2) Audio only devices (MP3 players)Audio only devices (MP3 players)
MPEG-2 (1994) A completely different standard from MPEG 1.A completely different standard from MPEG 1. Objective:Objective:
Deliver video and audio at comparable quality to Deliver video and audio at comparable quality to broadcast video at roughly 4 times the typical MPEG-broadcast video at roughly 4 times the typical MPEG-1 bit rate1 bit rate
Requires between 3 and 15 MbpsRequires between 3 and 15 Mbps Improvements over MPEG-1Improvements over MPEG-1
Higher resolutionHigher resolution Higher image qualityHigher image quality Supports interlaced video formatsSupports interlaced video formats Multiresolution scalabilityMultiresolution scalability Multiple sound track featuresMultiple sound track features ScaleableScaleable
The MPEG-2 Standard ISO 13818 – 10 partsISO 13818 – 10 parts 640 patents, held by 24 corporations and one university640 patents, held by 24 corporations and one university Video Encoding (grossly simplified)Video Encoding (grossly simplified)
I, P and B frames – similar to MPEG-1I, P and B frames – similar to MPEG-1 Video is separated into:Video is separated into:
1 luminance and 2 chrominance channels, 1 luminance and 2 chrominance channels, 4 luminance macroblocks 4 luminance macroblocks variable number of chrominance macroblocksvariable number of chrominance macroblocks
Macroblocks of P and B frames are compared, and a motion Macroblocks of P and B frames are compared, and a motion vector calculatedvector calculated
Each block is treated with DCT, the result is run-length Each block is treated with DCT, the result is run-length encoded and a huffman compression is appliedencoded and a huffman compression is applied
Scaleable MPEG MPEG-2 is scaleable (layered coding)MPEG-2 is scaleable (layered coding) Single coded bitstream is scaleable to various bitratesSingle coded bitstream is scaleable to various bitrates Layers can be incrementally decodedLayers can be incrementally decoded
Basic layer – low quality videoBasic layer – low quality video Subsequent layers – higher quality videoSubsequent layers – higher quality video
If the bandwidth is low, only the bitstream from the If the bandwidth is low, only the bitstream from the base layer is sentbase layer is sent
If the bandwidth changes, more or fewer layers may If the bandwidth changes, more or fewer layers may be usedbe used
The same principle can be used on noisy connectionsThe same principle can be used on noisy connections
Applications of MPEG-2 DVD VideoDVD Video HDTVHDTV Ditgital satelite / cable TVDitgital satelite / cable TV Digital video recordersDigital video recorders Video conferencingVideo conferencing Some gamesSome games Some multimedia (usually where very high Some multimedia (usually where very high
quality is essential)quality is essential) NB encoding is processor intensive – thus more NB encoding is processor intensive – thus more
expensive than MPEG-1expensive than MPEG-1
Editing MPEG 1 & 2 High performance compression formatsHigh performance compression formats
Not designed for editingNot designed for editing Only I-frames are stored in their entiretyOnly I-frames are stored in their entirety
This makes editing, rewinding or even bookmarking This makes editing, rewinding or even bookmarking extremely difficultextremely difficult
MPEG editors tend to be crudeMPEG editors tend to be crude Serious editing involves decmopressing and Serious editing involves decmopressing and
recompressingrecompressing Quality lossQuality loss
Moral – don’t edit MPEG if at all possibleMoral – don’t edit MPEG if at all possible Therefore don’t store digital video as MPEG!Therefore don’t store digital video as MPEG!
MPEG-4 (1998) A different approach to videoA different approach to video Designed primarily for streaming video (e.g. on WWW), Designed primarily for streaming video (e.g. on WWW),
3G phones, and other multimedia.3G phones, and other multimedia. MPEG-4 absorbs most of MPEG-1 and MPEG-2MPEG-4 absorbs most of MPEG-1 and MPEG-2 Adds: Adds:
VRML (3D) supportVRML (3D) support Object-based codingObject-based coding User interactivityUser interactivity Hypermedia capabilityHypermedia capability Streams from different sources can be combinedStreams from different sources can be combined Supports uploading and downloadingSupports uploading and downloading Rights management protocolRights management protocol
Implementation of MPEG-4 Designed to be modularDesigned to be modular
Standard in 20 parts (levels)Standard in 20 parts (levels) Each level contains profilesEach level contains profiles Developers implement components of MPEG-4Developers implement components of MPEG-4 Probably no full implementationProbably no full implementation
Various partial implementations, such as:Various partial implementations, such as: DivXDivX Microsoft ASFMicrosoft ASF Quicktime 6Quicktime 6
Object-based coding Media objectsMedia objects Audio and visual objectsAudio and visual objects
Natural objects (captured by camera)Natural objects (captured by camera) Synthetic objects (created by computer)Synthetic objects (created by computer)
Visual objects can be any shape (not just rectangles)Visual objects can be any shape (not just rectangles) Video manipulation (insertion, deletion, rotation etc.)Video manipulation (insertion, deletion, rotation etc.) Video indexing and retrievalVideo indexing and retrieval Texture mapsTexture maps SpritesSprites 3-d models and animation3-d models and animation
Probably the future of multimedia!Probably the future of multimedia!
MPEG-7 Multimedia content description interfaceMultimedia content description interface Aimed at addressing issues of content retrievalAimed at addressing issues of content retrieval
Intended primarily for digital librariesIntended primarily for digital libraries Object based (as MPEG-4)Object based (as MPEG-4) Supported data:Supported data:
Still imagesStill images 3D models3D models Sampled audioSampled audio Speech Speech VideoVideo Composition information (combinations of the above)Composition information (combinations of the above)
Data representationData representation Binary objects, XML or bothBinary objects, XML or both
MPEG-21 Designed to manage digital media resourcesDesigned to manage digital media resources
Digital item declarationDigital item declaration Digital item identification and descriptionDigital item identification and description Content management and usageContent management and usage Intellectual property management and protectionIntellectual property management and protection Terminals and networks Terminals and networks Content representationContent representation Event reportingEvent reporting