Introduction Dr Paul Vickers 1

School of Informatics

CG087 Time-based Multimedia Assets

Introduction to time-based multimedia assets

Dr Paul VickersDr Alf Watson

Introduction Dr Paul Vickers 2

CG087 Time-based Multimedia AssetsSchool of Informatics

Introduction

• Looking at multimedia assets that change over time– Sound– Video

• Sound clearly requires a time dimension to be perceived

• By video we mean either motion video, or sequences of still images that are time-based (e.g. synchronised slide show) or combination of both

Introduction Dr Paul Vickers 3

CG087 Time-based Multimedia AssetsSchool of Informatics

What the module covers

• Sound theory– Physics of sound– Psycho-acoustics

• Sound language– Atmospheric sound– Sonic/musical grammars– Foley art and sound tracks– Sound in games

Introduction Dr Paul Vickers 4

CG087 Time-based Multimedia AssetsSchool of Informatics

What the module covers

• Sound as a communication/interaction medium– Auditory display– Sonification, auralisation– Earcons, auditory icons

• Musical instrument digital interface (MIDI)– Definition & history– Protocols– Input– Controllers & tone generators– Sequencers

Introduction Dr Paul Vickers 5

CG087 Time-based Multimedia AssetsSchool of Informatics

What the module covers

• Combining MIDI and audio– Sampling & loops– Sequencing & mixing

• (Compression and streaming– What is compression?– Types of compression (e.g. WMA, MP3, ATRAC))

Introduction Dr Paul Vickers 6

CG087 Time-based Multimedia AssetsSchool of Informatics

What the module covers

• Video– Digital vs analogue– Filming – video language, managing shooting– Digitising, optimising, compression systems– Synchronising sound to video– Compression & streaming– Time-based control structures

Introduction Dr Paul Vickers 7

CG087 Time-based Multimedia AssetsSchool of Informatics

Who does what

• Dr Paul Vickers: Sound– paul.vickers@northumbria.ac.uk– Pandon 1.21– Tel 243-7614

• Dr Alf Watson: Video– alf.watson@northumbria.ac.uk– Pandon 1.14– Tel 243-7606

Introduction Dr Paul Vickers 8

CG087 Time-based Multimedia AssetsSchool of Informatics

Assessment

• There is one piece of assessment (no exam)– An asset creation, reporting and discussion task– Will be handed out in week 1– Due in week 14

Introduction Dr Paul Vickers 9

CG087 Time-based Multimedia AssetsSchool of Informatics

Introduction to sound

• What is sound?– If a tree falls in a forest and nobody is there to

hear it, does it make a sound?– Discuss…

• Different from vision– Can only attend to one visual stream at once, yet

can monitor several auditory streams– When you mix sounds from several sources we

still perceive them as separate sources

Introduction Dr Paul Vickers 10

CG087 Time-based Multimedia AssetsSchool of Informatics

Sound perception

• Sound is a construction of the mind• Neural coding & processing of information

from auditory system, integration with information from other sensory systems, and responding to the result is what defines hearing

• Sound is temporal– (what is a sound of zero duration?)– Though it has some spatial characteristics.

Discuss.

Introduction Dr Paul Vickers 11

CG087 Time-based Multimedia AssetsSchool of Informatics

Physical definition

• Vibration of an object produces sound. If we hear the vibration, the sound is audible– Hitting a table causes vibration– Blowing across a bottle causes complex vibrations– Plucking a guitar string causes it to vibrate

• What’s the difference between the sounds made by hitting a table, blowing a trombone, whistling?

• Fourier showed that any vibration can be resolved into a sum of sinusoidal vibrations

Introduction Dr Paul Vickers 12

CG087 Time-based Multimedia AssetsSchool of Informatics

Sinusoids

• Sinusoids (aka sine waves) describe relationships between displacement and time

– http://www.keypress.com/sketchpad/java_gsp/sinewaver.html

Introduction Dr Paul Vickers 13

CG087 Time-based Multimedia AssetsSchool of Informatics

Attributes of sine wave

• Frequency (no. of cycles per second, Hz)• Amplitude (height of wave)• Starting phase

– http://www.udel.edu/idsardi/sinewave/sinewave.html

• What does frequency relate to?• What does amplitude relate to?

Introduction Dr Paul Vickers 14

CG087 Time-based Multimedia AssetsSchool of Informatics

Attributes of sound

• Sound has three physical attributes– Frequency– Intensity– Time

• In music we talk about pitch. Is it the same as frequency?– Concert pitch is where the A above middle C has

a frequency of 440Hz

Introduction Dr Paul Vickers 15

CG087 Time-based Multimedia AssetsSchool of Informatics

Complex sounds

• Most sounds are not pure tones, but combinations of frequencies

• Pure tone:

Introduction Dr Paul Vickers 16

CG087 Time-based Multimedia AssetsSchool of Informatics

Complex sounds

• Square wave

• Triangle wave

Introduction Dr Paul Vickers 17

CG087 Time-based Multimedia AssetsSchool of Informatics

Pitch perception

• All three waves had the same perceived frequency yet the first contained only one sinusoid, whilst the square and triangle waves contained many lower-intensity sinusoids above the fundamental of 250Hz

• Thus, we tend to perceive the pitch of a tone as its fundamental (or its missing fundamental!)– You hear a 100Hz pitch when presented with a stimulus

consisting of the sum of the frequencies of 700, 800, 900, & 1000Hz: all four tones are harmonics of 100Hz

• Pitch of sine waves varies with intensity! Pitch increases as level increases

Introduction Dr Paul Vickers 18

CG087 Time-based Multimedia AssetsSchool of Informatics

Timbre

• What differentiated the sound of the three tones we heard earlier? How would you describe them?

• The quality that differentiates how tones sound is called timbre

• Different instruments have different timbres

Introduction Dr Paul Vickers 19

CG087 Time-based Multimedia AssetsSchool of Informatics

Intensity

• Intensity is an measure of the energy of a signal measured in decibels (dB)– 0dB is the threshold of human hearing

(0.000000000001 W/m2)– 3dB is a doubling of intensity– Pain threshold 120dB-140dB– 90dB (or prolonged exposure to lower levels) can

cause permanent hearing damage

Introduction Dr Paul Vickers 20

CG087 Time-based Multimedia AssetsSchool of Informatics

Power ratios and decibels

P/Pr Ratio expressed

in decibels 1,000,000 60 dB

10,000 40 dB 1,000 30 dB

100 20 dB 10 10 dB 4 6 dB 1 0 dB

¼ (=0.25) -6 dB 1/10 (=0.1) -10 dB

1/100 (=0.01) -20 dB P is the power of the measured sound in watts per square metre Pr is the reference-level power, or 10-12 watts-per-square-metre. Number of decibels is 10 log10(P/Pr)

Introduction Dr Paul Vickers 21

CG087 Time-based Multimedia AssetsSchool of Informatics

Example intensities

Description Intensity

Threshold of hearing 0 dB

Outdoors, no people or traffic 20 dB

Background noise in a library 40 dB

Ordinary speaking voice 60 dB

Heavy traffic 85 dB

Power lawnmower up close 90 dB

Motorcycle (rider) 90 dB

Chainsaw 100 dB

Rock music at concert 110 dB

Jet plane takeoff at 100 yards 120 dB

Threshold of pain 140 dB

Introduction Dr Paul Vickers 22

CG087 Time-based Multimedia AssetsSchool of Informatics

Loudness

• Loudness is a psychophysical response which does not necessarily equate to intensity. Usually measured in phons– Phon is the level in dB SPL of an equally loud

1,000Hz tone. All tones judged equal in loudness to a 40dB SPL 1KHz tone have a loudness level of 40 phons.

• Studies have shown loudness doubles every 10dB or so, i.e. sound must be increased in intensity by a factor of ten for the sound to be perceived as twice as loud. Or, it takes 10 violins to sound twice as loud as one violin.

Introduction Dr Paul Vickers 23

CG087 Time-based Multimedia AssetsSchool of Informatics

Equal loudness contours

100Hz, 52dB SPL1KHz, 40 dB SPL4KHz, 37 dB SPLAll judged to be

equal in loudness

Introduction Dr Paul Vickers 24

CG087 Time-based Multimedia AssetsSchool of Informatics

Psychoacoustics

• Psychoacoustics is the psychological study of hearing. The aim of psychoacoustic research is to find out how hearing works.

• We have seen that what we hear is not the same as what is presented

• Why do we hear things in certain ways?

Introduction Dr Paul Vickers 25

CG087 Time-based Multimedia AssetsSchool of Informatics

Auditory scene analysis

• “The things that we see are organized into patterns or figures, rather than discrete dots of light of different colours. In hearing, we also tend to organize sounds into auditory objects or streams. Bregman (1994) has termed this process for audition auditory scene analysis”– Taken from

http://privatewww.essex.ac.uk/~cplack/Courses/PS454_hbk.html

• For info on ASA see http://www.psych.mcgill.ca/labs/auditory/introASA.html

Introduction Dr Paul Vickers 26

CG087 Time-based Multimedia AssetsSchool of Informatics

Other phenomena

• The cocktail party effect (Arons, 1992) allows us to pick out one conversation in a babel of voices

• The tritone paradox (Deutsch, 1991) shows how our language influences the way we hear music– http://www.cameron.edu/~lloydd/webdoc1.html#SECTION3

• Auditory illusions– Shepard risset tones

http://www.exploratorium.edu/exhibits/highest_note/ex.about.fr.html

– And many others. See what you can find.

Introduction Dr Paul Vickers 27

CG087 Time-based Multimedia AssetsSchool of Informatics

References

• Arons, B. (1992). “A Review of the Cocktail Party Effect.” Journal of the American Voice I/O Society 12(Jul.): 35-50.

• Bregman, Albert S. (1994). Auditory Scene Analysis: The Perceptual Organization of sound. Cambridge, Massachusetts: The MIT Press.

• Deutsch, D. (1991). “The Tritone Paradox: An Influence of Language on Music Perception.” Music Perception 8(4): 335-347.