T325: Technologies for digital media

T325 - Technologies for digital Media

T325: Technologies for digital media Second semester 2011/2012Tutorial 4 Seeing and Hearing: Multimedia Output and InputArab Open University Spring 2012 11OutlineIntroduction PerceptionDisplay technologiesRasters and video signalsGraphics operationsCapturing the contentThe sound channel

Arab Open University Spring 2012 22IntroductionHow we actually see things ?How display data for real and virtual images is obtained and manipulated?Technologies involved in presenting multi-media informationTechnologies involved in capturing it from the outside world or conjuring it up from virtual objects in a computerSound channels

Arab Open University Spring 2012 3PerceptionArab Open University Spring 2012 4QuestionWhy do you hear by perception ?

Why do you think perception is important for multimedia technologies?Arab Open University Spring 2012 5IntroductionPerception has been of interest to technologists from the earliest days of radio and television, largely because of the bandwidth issue.Studies of human perception indicate what is important to send, and what can safely be discarded.Human perception theory is an essential factor in communication systems standardsArab Open University Spring 2012 6Light and ColorLight is a form of electromagnetic radiation, like radio but at a much higher frequencythe colors of the rainbow are due to light at specific wavelengthsA color TV picture is made up of red, green and blue dots (RGB) (additive colors) and a color printer uses yellow, cyan, magenta and black (CYMK) (subtractive colors) inks.There is no need to send data representing the entire range of visible wavelengths for each point in a scene; just three quantities will do.

Arab Open University Spring 2012 7

Source of the images : www.wikipedia.com7Light and Color - ActivityThe wavelength of visible light is between about 400nm and 700nm (1nm = 10-9 m). If the speed of light is 3 x108 m/s, what is the frequency range of visible light? (Frequency = speed/wavelength.)AnswerAt 700 nm the frequency is (3 x108)/(700 x10-9) Hz, which is approximately 430 x1012 Hz or 430 THz (terahertz).At 400 nm the frequency is :(3 x108)/(400 x109) Hz = 750 x1012 Hz or 750 THz.Arab Open University Spring 2012 8Light and colorThe eye as a camera has a light-sensitive surface, the retina, analogous to the film in a film camera or the detector in a digital camera.A lens forms an image on the retina.The iris, like a camera aperture, responds to sudden changes in brightness The retina has specialized sensory cells which convert light to electrical signals by triggering nerve impulses


9Light and ColorNerves carry signals in the form of trains of impulses to the regions of the brain that deal with sensory input. Impulses are all or nothing; there are no half-size impulses. There is a time gap between impulses there is a maximum rate at which nerves can conduct impulses.Brain Processing : timescales of milliseconds Silicon logic : nanosecond timescalesVast numbers of interconnections between individual nerve cells or neurons makes the brain particularly adept at parallel processing operations: pattern recognition, that have been found difficult to implement on traditional computers with their serial architectures.Arab Open University Spring 2012 10Light and colorThere are four types of light-sensitive cell in the retina: S cones, which respond preferentially to shorter wavelength light.M cones, which respond preferentially to medium wavelengths.L cones, which respond preferentially to longer wavelengths.Rods, which are very sensitive cells used in night vision. Arab Open University Spring 2012 11

References for figures: http://users.rcn.com/jkimball.ma.ultranet/BiologyPages/V/Vision.htmlhttp://www.oneminuteastronomer.com/wp-content/uploads/2009/06/rods_cones.gif11Light and ColorColor decision (eye and brain) relative responses of the three types of cone.Displays with RGB dots stimulate L, M and S cones respectively can give the illusion of a wide range of colors simply by varying the relative brightness of the dots


Responses of L, M and S cells Light and color Cones do not respond instantly when the light falling on them changes.The chain of chemical reactions behind their operation is very fast, but still takes timeA flash of light causes a response in a cone that builds to a maximum in about 50 ms and lasts about 200 ms, so images on the retina persist for a short time. This persistence of vision is relevant to television and film, where sequences of still images are perceived as smooth motionArab Open University Spring 2012 13Luminance &ChromaticityAlternative to the RGB representationA light source can be characterized by its: Luminance: a measure of how bright it isChromaticity: which identifies the particular shade of color Shining a white light on an object increases its luminance without significantly affecting its chromaticity. Luminance is related to the total of the red, green and blue componentsChromaticity depends on their relative proportions.Arab Open University Spring 2012 14Luminance is related to the total of the red, green and blue components, while chromaticity depends on their relative proportions

14CIE chromaticity diagramThe points along the curved boundary correspond to light of single wavelengths (the rainbow spectrum).There is a point in the middle of the diagram that corresponds to light of all wavelengths (perceived as white, or grey at lower levels of luminance).The diagram shows one particular level of luminance. You can imagine a series of such diagrams all similar but lighter or darker to cover the full range of possibilities.


CIE chromaticity diagramThe triangle shown corresponds to the colors that are available on a particular RGB display. The three corners of the triangle represent the red, green and blue primary colors of the display. Any color within the triangle or along its edges can be reproduced by the display, using red, green and blue in the appropriate proportions


CIE chromaticity diagramSome colors lie outside the triangle cannot be reproduced accurately with this display. You would see a nearby color from the range that the display is capable of showing.The range of colors available on a display is called its gamut. Color printing also has a gamut. Generally, the gamut does not cover the entire CIE chromaticity diagram Colors outside the gamut are displayed as a near equivalent displayable color. Arab Open University Spring 2012 17Color in TV and videoThe human visual system is much better at resolving luminance information than chromaticity or chrominance.The analogue system has a full-bandwidth luminance signal together with two chrominance signals with much lower bandwidth.RGB signals, such as those from a color TV camera, can easily be converted to a luminance signal and two chrominance signals.The eye is most sensitive to green, less so to red and less again to blue the components are given unequal weighting. The luminance signal Y is given by : Y = 0.587G + 0.299R + 0.114B The chrominance signals are also derived by combining the R, G and B signals. U = 0.492(B - Y) and V = 0.877(R - Y)Note: the formulas dont need to be memorized

Arab Open University Spring 2012 18The luminance signal is designed to be effectively the same as the signal from a monochrome camera pointed at the same scene, so that a monochrome display can simply use the luminance signal and discard the chrominance signals.

18Depth in imagesPhotographs, paintings and TV pictures give us a two-dimensional representation of a three-dimensional world. Our two eyes provide slightly different viewpoints of a three-dimensional scene, and this gives some clues about depth. But we can still appreciate the depth in a picture with one eye closed.There are several visual cues in an image that give us this feeling of depth. They come across automatically when it is a picture of a scene taken with a camera.Should be taken into consideration when creating a three-dimensional object in a computer that will look correct on a two-dimensional screen just as a painter does

Arab Open University Spring 2012 19Depth in imagesArab Open University Spring 2012 20

Display TechnologiesArab Open University Spring 2012 21IntroductionA display is an array of red, green and blue elements Each element can individually be controlled in brightness so as to convert images from digital data to visible form.


Classification of Display techniquesTransmissive methods: where the picture elements pass or obstruct light. The light energy comes from a light source behind the array of picture elements. Example: Liquid-Crystal display (LCD)Emissive methods: where each picture element emits light, through the conversion of electrical energy. Example: Cathode ray tube (CRT) and PlasmaReflective methods: where the picture elements reflect more or less of the light that shines on them and thus appear light or dark. The light energy comes entirely from ambient lighting. Example: Electronic paper Arab Open University Spring 2012 23Light Emitting Diode (LED)Light-emitting diodes (LEDs) are semiconductor devices that convert electricity directly and efficiently into light.The wavelength of light emitted is determined by the material used. Red, green and blue types are available and can be put together in an array to make a display. Very large emissive displays of this type are often seen at entertainment and sporting events.Arab Open University Spring 2012 24

Display technologies Activity 4.5How many LEDs would be needed to make a 640 x 480 pixel screen of this type, if a pixel consists of one LED of each color?640x480 = 307 200 of each of red, green and blue or3 x 307 200 = 921600 LEDs in total.

Assuming each LED occupies 1 cm2, what area would it cover?1m2 = 10^4cm^2, so the area is 921600/10^4 = 92 m2 approximately

Arab Open University Spring 2012 25Liquid-Crystal Display (LCD)Liquid-crystal displays (LCDs) operate by controlling the passage of light. Most LCDs (used television and computer monitors) are illuminated from behind with a uniform artificial backlightSome types are viewable using only ambient light, by means of a reflective layer behind the displayConsume more power than LEDs: Although LCDs consume little power themselves, but the backlight needs power and this can be a consideration with battery equipment.Arab Open University Spring 2012 26LCDs can also be used in projection applications, where a relatively small LCD controls the light projected on the front or back of a much larger screen.

26Liquid-crystal displays LCDs depend on polarization of light, a property which is exploited in many sunglasses.Light is a transverse wave, as with all types of electromagnetic radiation. Electromagnetic wave: electric field component and a magnetic field component at right angles to the direction of travel and also at right angles to each other. Arab Open University Spring 2012 27

The light from glare that sunglasses are designed to reduce is often polarized in the direction that the lenses do not transmit.27Liquid-crystal displays If the fields are predominantly in one direction then the radiation is said to be polarised.Example: if the electric field is up and down in a wave moving horizontally Many radio and TV transmitters use polarised waves, which result from the design of the antenna.Most light sources are polarized, sending out waves of all orientations. Polarising filters such as those used in sunglasses pass only light which is oriented in a particular direction. Arab Open University Spring 2012 28

Liquid Crystal DisplaysLight is twisted using liquid crystalsLiquid crystals are unusual substances. The molecules move around quite freelyThey do not have a fixed long-term structure like crystalline solids.


Source : http://static.howstuffworks.com/flash/lcd-twisted.swf

29Liquid Crystal DisplaysThe molecules tend to align themselves in loosely ordered structures. The molecules tend to line up in an electric field.At and around surfaces, the molecules tend to follow features such as parallel grooves This gives some control over the structure of the liquid crystal. The ordered molecules can affect the direction of polarization of light.

Arab Open University Spring 2012 30Liquid Crystal DisplaysLiquid crystals used in displays are held in a thin layer between two glass sheets.In one common type, the two glass sheets are covered with a polymer surface with parallel grooves. The grooves on one sheet are at right angles to the grooves on the other sheet. This imposes a twisted structure on the liquid crystal molecules.While the liquid crystal has this twisted structure, light shining through it has its polarization direction rotated through 90

Arab Open University Spring 2012 31Liquid Crystal DisplaysThe assembly of liquid crystal held between the two glass sheets is placed between two polarizing filters.Because of the twist, light can go through both filters. So we have reversed the action of the filters. To be useful as a display, though, we also need a way of returning it to the previous condition of stopping light. This is simply done by applying an electric field, which lines up the liquid crystal molecules in a parallel arrangement which does not affect polarization.Arab Open University Spring 2012 32QuestionHow to control each element (pixel) on a LCD TV screen?Arab Open University Spring 2012 33Liquid Crystal DisplaysMatrix arrangement of rows and columns individual pixels could be addressed by selecting the correct row and columnExample: an array of 1000 x 1000 pixels would only need 2000 conductors rather than a million.The top electrodes of each column are connected together, and so are the bottom electrodes of each row.Addressing a particular pixel apply a voltage between the conductors for the pixels row and column.


Liquid Crystal DisplaysA device is needed at each intersection which will respond when the correct row and column signals are applied but not otherwise Use thin-film transistor (TFT).A TFT, like other transistors, can be used as a semiconductor switch, turning on and off the voltage across the LCD elements.


Plasma displaysPlasma displays fall in the emissive category. Use similar physical principles to fluorescent lighting.A display is made up of an array of small cells that can be controlled individually.A plasma is a mixture of free electrons, atoms from which an electron has been removed (ions) which have a positive charge, and unaffected atoms can be caused by very high temperatures or by electric fieldsPlasmas are produced in a special gas mixture in fluorescent tubes by applying a voltage between the two ends. Arab Open University Spring 2012 36Plasma displaysFreely moving electrons in plasma can carry an electric current. Energy supplied to the plasma from the current source is turned into radiation at certain specific wavelengths. This radiation may fall in the visible regionIn practical, plasma radiation is converted to a different wavelength using a phosphor because much of the radiation from effective plasma sources is in the invisible UV region.Arab Open University Spring 2012 37Plasma displaysPhosphor example : The white material that coats the inside surface of a fluorescent tube A phosphor absorbs energy and then reradiates it at fixed visible wavelengths. There is a well-established technology for producing efficient phosphors with bright colors. The phosphor used in fluorescent tubes emits white light and phosphors in CRTs emit red, green and blue. In CRTs, the energy is supplied to the phosphor by an electron beam rather than by UV radiation, but the effect is the sameArab Open University Spring 2012 38Electronic paperElectronic paper has been proposed as a reflective type of display Visually more similar to reading print on paper than looking at a conventional computer screen. Being reflective rather than backlit, they are thought to be easier on the eye. Non-volatility feature: the image does not disappear when the power is switched off.Some of the technologies use floating charged particles with sizes in the order of tens of micrometers, suspended between two flexible sheets. The sheets have transparent electrode arrays which can attract or repel the particles, bringing them into or out of view.


Rasters and Video SignalsArab Open University Spring 2012 40Rasters and video signalsFor TV, the picture arrives as a serial stream of analogue or digital information, in a standard format where exact timing is important.The end product is three video signals representing red, green and blue, together with signals that are used for synchronization.When computer monitors were introduced they followed TV practiceArab Open University Spring 2012 41Rasters and video signalsIn digital systems the line is a row of a finite number of pixels; for example, 720 pixels/line.Arab Open University Spring 2012 42

Moving images are an illusion produced by displaying a series of still images or frames. TV frames are built up by scanning: writing a line at a time and moving down to the next line when a line is complete, in the same way that we read text on a page.The lines were created by scanning, thus covering the whole screen with a pattern called a raster. Rasters and video signalsThe idea of raster scanning is deeply embedded in computer graphics. Image data is routinely held in blocks of memory in the expectation that it will be read out in a raster scanned fashion.In a computer, the video output is produced by repeatedly reading out the contents of a block of memory (sometimes called the frame buffer) and sending it to a digital-to-analogue converter.Each memory location holds a coded value for the red, blue and green content of a pixel.

Arab Open University Spring 2012 43Rasters and video signalsWith the demand for higher definition and a greater range of colors, the volume of data that needs to be sent to the display has increased dramaticallyThis led to introduction of new digital interface standards to replace the traditional analogue interface.A high-definition TV data stream compressed for broadcast takes 19 Mbit/s, while the data rate off a Blu-ray disc is 48 Mbit/s.HDMI1.3 specification offers 10.2 Gbit/s of bandwidth, to support a 48-bit rather than 24-bit color gamutLast version of HDMI specification is 1.4a (March 2010)

Arab Open University Spring 2012 44Graphics operationArab Open University Spring 2012 45Two dimensional operationsA picture can be built up by reading out the contents of a memory, with a one-to-one correspondence between memory locations and the pixels on a screen.A two dimensional operations to move objects around the screen can be done just by changing where the data representing the object is in the memory.

(See Activity 4.10)

Arab Open University Spring 2012 46Graphics operations

Arab Open University Spring 2012 47If the location contains the data 255 the pixel is fully bright, and if it contains 0 the pixel is fully dark. The data shown will display a triangle.

47Two-dimensional operationsAn object may be moved around the screen (translation) by simple addition of memory addresses. Related techniques can be used to perform other operations, such as scaling (expansion or contraction) or rotation.If there are a number of objects moving around, they may need to pass in front of or behind each other.Remember that interposition or occlusion is an important factor in how things are perceived.

Arab Open University Spring 2012 48So an object can be moved around, but what if the background is not simply blank, but is some type of image?

48Two-dimensional operationsSpeeding up two-dimensional operations by processing a whole block of data (in data transfer and logical operations) instead of a byte or word at a time Introducing new set of computer instructions : bit block transfers (abbreviated to bitBLT or simply BLT).Use of dedicated hardware in the graphics processors for BLT to relieve the main processor of this burdensome taskThese operations are simple and repetitive but involve moving large amounts of data, Carrying them out separately from the main processor can save a significant amount of computing time.

Arab Open University Spring 2012 49It was realized that these types of two-dimensional operation could be carried out more quickly if, instead of performing the data transfer and logical operations a byte a whole block of data couldbe processed together.

49Three-dimensional operationsMuch of the early interest in three-dimensional graphics was related to CAD (Computer-aided design).A designer could create a virtual three-dimensional model in a computer, display it on a screen and look at it from all angles.One of the first problems to be solved was how to represent a three dimensional object without needing to store and process an unfeasibly large mass of data. Most three-dimensional objects could be represented by storing the coordinates of a relatively small number of points. The rest of the object could be filled in later using a variety of algorithms to interpolate between the known points.

Arab Open University Spring 2012 50You have seen that two-dimensional images can make heavy demands even on modern computers; adding a third dimension could increase the problem by orders of magnitude.50Three-dimensional operationsA commonly used representation is the wireframe, where a set of points, or vertices (singular vertex), are joined by straight lines.A wireframe alone is sufficient to visualize the object, but usually a wireframe is a conceptual skeleton which is completed by rendering to create a solid looking object with filled in surfaces.Now any point (or vertex) can be located in three-dimensional space by a set of three numbers, called a vector.An object described by a set of vectors can be translated, scaled or rotated by multiplying the vectors by a matrix to give a new set of vectors.

Arab Open University Spring 2012 51Three-dimensional operationsWhat you see depends on what angle you view the object from; indeed, one of the most attractive features of three-dimensional computer graphics is that objects can easily be viewed from any angle. Surfaces that normally cannot be seen need to be removed.Lighting provides the shading cues that are so important for depth perception.One of the most computationally intensive tasks, and one that contributes most to the perception of realism, is to determine how much light reflects off every point of the objects visible surface.

Arab Open University Spring 2012 52Three-dimensional operations

Arab Open University Spring 2012 53Graphics processorsGraphics processors, otherwise known as graphics processing units (GPUs) or graphics accelerators, are designed specifically to carry graphics-related calculations (especially 3D)The graphics processor is closely connected to the display or video interface.It usually has its own memory, although some share the computers main memory.


Capturing the contentArab Open University Spring 2012 55Capturing the contentDigital images more usually originate in a camera of some sort.The optical components of a digital camera are similar to those of a traditional film camera.The image is formed on a microelectronic sensor.The sensor is an array of individual light-sensitive pixels. Most cameras use a sensor of one of two types: CCD (Charge Coupled Device) and CMOS (Complementary Metal Oxide Semiconductor).The main difference between the two lies in the way that data is read out.

Arab Open University Spring 2012 56Motion captureA computer-generated image could start off from just a few points and then fill in the gaps to create a solid-looking image.Motion capture in a way does the opposite it captures the motion of a relatively few points from a real person or object.One type of motion-capture system uses small reflecting marker balls that are stuck on the points of interest. These marker balls are illuminated with infrared light which reflects from them and is picked up by one or more of a set of cameras.

Arab Open University Spring 2012 57Motion CaptureOnce the motion-capture data has been collected, the observer can visualize the motion from all possible angles.Further, the data can be analyzed and manipulated in greater depth; for example, distances and angles between the marker balls can be measured at any point in the motion.


The sound channelArab Open University Spring 2012 59The sound channelMicrophones : A microphone transforms sound waves in air to an electrical signal. Any process that converts mechanical movements to electrical energy can potentially form the basis of a microphone.Many microphones make use of Faradays law of magnetic induction

Arab Open University Spring 2012 60The sound channel

Arab Open University Spring 2012 61The ribbon is a light, flexible electrical conductor. It is placed in a strong magnetic field between the poles of a permanent magnet. When it is moved backwards and forwards by a sound wave, a voltage is induced which varies with the velocity of movement.Directional microphonesRibbon microphone

LoudspeakersA loudspeaker, or speaker for short, performs the converse operation to a microphone, converting electrical energy to sound waves.It does this by moving a stiff cone backwards and forwards, together with the air surrounding it.


LoudspeakersPhysical limitationsLarge diameter cones are effective at moving large volumes of air and are good at reproducing low bass notes,their mass makes them perform less well with high notes.How to solve this?two or more speakers, optimized for different frequency ranges, are used in the same enclosure.Arab Open University Spring 2012 63The cone needs to be stiff and light at the same time;63Sound localization and stereoThe sound channel, like vision, draws heavily on our knowledge of human perception.Sound travels at a much lower speed than light (340 m/s)Two possible ways of locating sounds are using:Time differences : a sound located to the hearers right will arrive at the right ear before the left earIntensity differences : less sound energy may reach one ear, particularly if the head is in the way.A sudden noise, like a click, will arrive at the nearest ear first, and it is not hard to see how the position of the source might be estimated by comparing the arrival times.The sound wave is a series of alternate peaks and troughs, so we need to look at the phase between the sound arriving at the two ears.Arab Open University Spring 2012 64Sound localization and stereoA peak arriving at one ear is compared with a corresponding peak arriving at the other ear.This works well over a range of frequencies, but breaks down when the wavelength is about the same as the distance between the ears


Sound localization and stereoWhen the wavelength is the same as the head diameter, so there is no phase difference between the sounds arriving at the two ears; the listener may perceive the sound as coming from directly in front, when in fact it is coming from the rightArab Open University Spring 2012 66

Sound localization and stereoPhase information is useful up to about 1500 Hz. Above this frequency, intensity differences are the main cue for sound location.As sound localization is based on comparisons of timing and intensity between the two ears, a quite realistic stereophonic effect can be obtained with just two speakers.Home cinema systems, which aim for good surround sound behind as well as in front of the listener, use additional speakers.In many home cinema systems just one large speaker, known as a subwoofer, The other speakers handle the rest of the frequency range and may be modest in size.

Arab Open University Spring 2012 67The lowest frequencies are difficult to localize in any case.The wavelength is much greater than the distance between the ears, so phase differences are only a small fraction of a wavelength and difficult to detect;

67

T325: Technologies for digital media

Documents

Transcript of T325: Technologies for digital media