Java_3D_stb

A Java 3D Framework for Digital Television Set-top Box

Master’s Thesis

Yongjun Zhang

Helsinki University of Technology Department of Computer Science and Engineering Telecommunications Software and Multimedia Laboratory Espoo, 15 October 2003

Yongjun Zhang : A Java 3D Framework for Digital Television Set-top Box

Abstract ii

ABSTRACT

Author: Yongjun Zhang University: Helsinki University of Technology Laboratory: Telecommunication Software and Multimedia Department: Computer Science and Engineering Professership: T-111 Interactive Digital Media Name: A Java 3D Framework for Digital Television Set-top Box Date: 15.10.2003 Num. Pages: 56 Supervisor: Petri Vuorimaa Instructor: Pablo Cesar Digital television offers a revolutionary platform for future home entertainment. Based on this platform, many innovative services and applications are deployed, presenting unprecedent user experience to TV viewers. The success of 3D graphics applications in PC industry also motivates hardware and software vendors to introduce 3D applications in digital television. A generic 3D framework is crucial to make 3D applications interoperable in different hardware and software platforms. Idealy, this framework will abstract all the platform-specific 3D operations, which makes it possible to develop applications that could be ‘written once and executed anywhere’. The aim of this thesis is to implement such a framework to facilitate the development of 3D applications in digital television. Java is a widely adopted in digital television industry, mainly because Java provides excellent platform-indenpendency for cross platform applications. TVGL, the 3D library developed in this thesis, is also implemented in Java. Java native interface technology is used to connect the high level Java API layer with the low-level native 3D libraries. In addition, the framework also proivdes a skeleton application for developers to start with, and two sample applications are demonstrated. 3D applications demands hugh computing resources, while the hardware capability of a digital set-top box, the device that executes services and applications, is very limited. To study the feasibility of 3D in digital television, a survey on the mid-range set-top boxes in the market is conducted. The feasibility study and demos show that 3D lightweight applications are possible with current set-top box configurations. As a new framework, TVGL can evolve to be a standardized API, making its way in extending the current digital television standards and enriching the profile sets of current digital set-top boxes.

Key Words: Digital Television, Set-top Box, OpenGL ES, Java Language: English


Abstract iii

ACKNOWLEDGEMENT

This thesis was written at the Telecommunications Software and Multimedia Laboratory, at Helsinki University of Technology, under the supervision of Professor Petri Vuorimaa.

I want to give my deepest thanks to Mr. Pablo Cesar, my instructor and tutor during the thesis work. Without his invaluable help and guidance, this thesis would not be possible.

I am very grateful to my supervisor Professor Petri Vuorimaa, for his advice and instructions, and for his vision towards my work, which encourages me to carry on the thesis.

Finally, I would like to present my gratitude to Mingfang Lai, my beloved wife, for her support at home.

Espoo, 15 October 2003

Yongjun Zhang


Table of Content iv

TABLE OF CONTENT

Abstract.................................................................................................................................... ii Acknowledgement .................................................................................................................. iii Table of Content ......................................................................................................................iv Abbreviations ..........................................................................................................................vi 1. Introduction ..........................................................................................................................1 2. Digital Television Technology .............................................................................................2

2.1 Introduction to DTV .......................................................................................................2 2.1.1 An Architecture View of DTV ................................................................................2 2.1.2 DTV Standards ........................................................................................................3 2.1.3 Adoption of DTV ....................................................................................................4

2.2 Digital Set-Top Box........................................................................................................5 2.2.1 STB Functionalities .................................................................................................5 2.2.2 STB Hardware Architecture ....................................................................................6 2.2.3 STB Software Platform ...........................................................................................9

2.3 DTV Applications and Services ...................................................................................11 2.3.1 Enhanced Information Services.............................................................................12 2.3.2 Enhanced Television .............................................................................................13 2.3.3 Interactive Services ...............................................................................................13

3. Java in digital set-top box...................................................................................................16 3.1 Introduction ..................................................................................................................16 3.2 Java Architecture ..........................................................................................................17

3.2.1 Java Virtual Machine.............................................................................................18 3.2.2 Java API.................................................................................................................19

3.3 Java in Digital STB.......................................................................................................20 3.3.1 Sun’s Java TV API ................................................................................................20 3.3.2 DAVIC Java Centric Solution ...............................................................................22 3.3.3 DVB-MHP.............................................................................................................22

4. 3D Graphics in Digital STB ...............................................................................................25 4.1 Introduction ..................................................................................................................25 4.2 An Overview of 3D Computer Graphics ......................................................................25

4.2.1 3D Rendering Pipeline ..........................................................................................25 4.2.2 3D APIs .................................................................................................................27

4.3 3D Graphics in Digital STB .........................................................................................28 4.3.1 3D Applications in Digital STB ............................................................................29 4.3.2 Feasibility Study of 3D Graphics in STB..............................................................31

5. Framework design and implementation .............................................................................33 5.1 Introduction ..................................................................................................................33 5.2 An Overview of the Reference STB Platform..............................................................33 5.3 Architecture Design and Implementation.....................................................................34

5.3.1 Java and 3D Rendering API ..................................................................................34 5.3.2 Java Native Interface .............................................................................................35

5.4 Future Improvements....................................................................................................38 5.5 Case Study ....................................................................................................................40

5.5.1 Demo Environment ...............................................................................................40 5.5.2 Demo Applications ................................................................................................41 5.5.3 Results Analysis ....................................................................................................42


Table of Content v

6. Conclusion..........................................................................................................................44 References ..............................................................................................................................45 Appendix A: Implemented OpenGL ES Functions................................................................49


Abbreviations vi

ABBREVIATIONS

API Application Programming Interface ARIB Association of Radio Industries and Business ATSC American Television Standards Committee CA Conditional Access CG Computer Graphics CI Common Interface COFDM Coded Orthogonal Frequency-Division Multiplex CPU Center Processing Unit DAVIC Digital Audio Video Concil DTV Digital Television DVB Digital Video Broadcasting ECM Entitlement Control Messages EMM Entitlement Management Messages FCC Federal Communication Committee FEC Forward Error Correction FPS Frames Per Second HAVi Home Audio Video interoperability HTTP Hyper Text Transfer Protocol ITC Independent Television Committee JIT Just-In-Time JMF Java Media Framework JNI Java Native Interface JVM Java Virtual Machine LAN Local Area Network MHP Multimedia Home Platform MMS Multimedia Message Service NVOD Near Video on Demand OFDM Orthogonal Frequency-Division Multiplex OO Object Oriented OSD On Screen Display PC Personal Computer PCMCIA Personal Computer Memory Card International Association PDA Personal Data Assistant PID Packet Identification PPV Pay Per View PSI Program Specific Information PSTN Public Switch Telephone Network PVR Personal Video Recorder


Abbreviations vii

QAM Quadrature Amplictude Modulation QPSK Quadrature Phase-Shift Keying Modulation RAM Random Access Memory RF Radio Frequency RMI Remote Method Invocation ROM Read Only Memory SMS Short Message Service STB Set-Top Box UI User Interface URL Unified Resource Location USB Universal Serial Bus VOD Video on Demand VRML Virtual Reality Markup Language VSB Vestigial Sideband XML eXtensible Markup Language


Introduction 1

1. INTRODUCTION

As the outcome of convergency of television and digital computing technology, Digital Television (DTV) is a revolutionary media platform for home entertainment. It benefits everyone including manufactures, content providers and end customers. Therefore, intensive researches are undergoing worldwidely to explore the opportunities brought by DTV. Interactive services and applications are one of the most interesting topics.

In PC, 3D graphics applications, like 3D games, have been very successful in attracting end-users by presenting unprecedent user experience, and a lot of business opportunities are created. DTV, with its growing popularity, would be an ideal 3D graphics platform for mass audience. However, technical restrictions limit the adoptation of 3D graphics to DTV platform and a lot of work needs to be done to this end.

This thesis aims to implement a Java based 3D Application Programming Interface (API) on digital Set-Top Box (STB), the device that receives DTV singals and excutes DTV services. Designing and implementing such an API in STB platform with limited hardware capability is quite challenging, because 3D graphics calculation is computational demanding and usually requires large amount of memory. Fortunately, STB hardware has been improving constantly and it is believed that STBs capable of doing realtime 3D graphics will eventually be common in the market.

Adopting 3D graphics to STB platform opens a door to a variety of new services because it extends the TV viewing experience from 2 dimensions to 3 dimensions. For example, local interactivity could be achieved easily with 3D content – viewers can interact with the scene after it is fully loaded; 3D widgets and controls could be used to improve the usability of DTV dramatically because they are more intuitive and user friendly. It is difficult to enumerate all the opportunities presented by 3D graphics in DTV environment.

The content of this thesis is divided into four parts. In the first part, the general DTV technology, including its architecture and adoption status, is introduced. STB as a receiver of DTV broadcasting signal is also discussed and various DTV services are introduced. The second part is focused on Java technology and its application in STB platform. Serveral DTV Java standard APIs are discussed in this part as well. 3D graphics is introduced in the third part, and a hardware feasibility study is undertaken to find out whether it is possible to carry out 3D applications in current STB platforms. The last part elaborates some technical issues emerging when designing and implementing the API, and two demo applications and their results are discussed. Finally, conclusions of the thesis work are given.


Digital Television Technology 2

2. DIGITAL TELEVISION TECHNOLOGY

2.1 Introduction to DTV

Many exciting new technologies, which have emerged during the last decade, are results of convergency between computing technology and a certain relatively traditional techonology; DTV is a good example of this. By combining computing with traditional video broadcasting, DTV provides unprecedent entertaining experiences to viewers, and thus great revenues to broadcasters. Analog broadcasting, for example, suffers from interference problem and picture quality deteriorates over distance and adverse reception condition [1]; this will however never be a problem in DTV. Moreover, digital compression in the process of content transmission squeezes more data into the same bandwidth, thus more channels and services could be packed within the same frequency resource, which is only enough for one analog TV channel.

In general, DTV technology digitizes the content in virtually every stage of television programme broadcasting, from content production and transmission to receiving. Production technology is already mainly digitized, including acquiring content using digital video camera, processing it with video editing software, and storing it in digital formats. Therefore, the term DTV usually refers to the digitization of content transmission and receiving stages.

2.1.1 An Architecture View of DTV

A brief architecture view of DTV system is illustrated in Figure 1, which is a modified version from Canal Digital [2]. In studio, the recorded video and audio contents are firstly processed and encoded into MPEG-2 flow. MPEG-2 is the video compression standard proposed by Motion Picture Export Group; it could provide significant compression ratio while preserving nice quality. Additional data, e.g., programme information, is delivered into the object carousel and then intermingled with the compressed audio/video data in the multiplexor, and a Conditional Access (CA) server provides information for encrypting to protect the programme from unauthorized viewing. The output of multiplexing is encrypted or

Audio/videoCompression

MultiplexorScrambler

ServiceDatabase

Data/Object Carousal

Conditional Access Server

Application servers

Transmissionmedium

Receiver

IP Network

Studio

TV set

Figure 1. A brief architecture view of DTV system [2].



scrambled MPEG-2 bit stream and ready for transmitting.

MPEG-2 bit stream can be transmitted by three means: through a terrestrial network, by cable and by satellite. For each transmission medium, the modulation method is different, and a dedicated receiver is needed to demodulate the signal. Besides demodulation, the functionalities of this receiver also include decompressing the MPEG-2 stream, splitting, decrambling and processing audio/video and data packets, and forwarding the signal to an ordinary TV-set.

An interaction channel could be established between the receiver and the servers of channel owner. The interaction channel is usually implemented using a telephone line with a dial-up modem, or using cable modem in cable network. The receiver collects the viewer’s input events through traditional remote controll or wireless keyboard. Some events are consumed by the receiver to perform local operations like selecting a channel or jumping to a particular teletext page. Some are forwarded to the server through IP network and the server will react accordingly. In this way, a full range of two-way interaction (i.e., between the content provider and user) services are possible.

2.1.2 DTV Standards

Currently, there are three different DTV standards coexisting in the world: American Television Standards Committee (ATSC) standard in USA, Digital Video Broadcasting (DVB) in Europe, and Association of Radio Industries and Business (ARIB) in Japan [3]. The rest counties mainly adopt one of these standards.

Being basically a terrestrial transmission standard, the ATSC defines the bit stream content and its digital transmission in a 6-MHZ bandwidth Radio Frequency (RF) channel [4]. Like DVB and ARIB, ATSC also uses MPEG-2 standard for video compression, and it supports up to 18 video transmission formats. Dolby AC-3 5.1 sound sysytem is applied as the audio format in ATSC, which encodes five full-bandwidth surrounding audio channels and one low-frequency enchancement channel. In addition, ATSC uses 8- Vestigial Sideband (VSB) modulation for terrestrial broadcasting, and 16-VSB for cable network broadcasting.

The European DVB standard defines three broadcasting specifications for different transmission medium: DVB-C for cable network, DVB-S for satellite and DVB-T for terrestrial transmission. The main difference between these standards is the modulation method applied; DVB-S is based on Quadrature Phase-Shift Keying Modulation (QPSK), whereas DVB-C uses Quadrature Amplictude Modulation (QAM) and DVB-T uses Coded Orthogonal Frequency-Division Multiplex (COFDM). Unlike ATSC, DVB group chose the MPEG-2 sound system, which is currently stereo only, but upgradable to multichannel surround. DVB also proposes a common interface for encryption, and broadcasters are free to select their own CA system. Video in DVB is based on standard resolution TV – 625-line/50Hz interlaced pictures, and doubles the resolution for high-definition mode. Despite of its maturity, DVB standard keeps improving and new standards are added continuously, like DVB-X for DTV broadcasting to mobile devices [5]. The Japan DTV standard, ARIB, also defines broadcasting standards for cable, satellite and terrestrial. However, the terrestrial specification in ARIB is distinguished from those of other nations by support for robust mobile reception. In addition, ARIB selected MPEG-2’s



Advanced Audio Coding (AAC) for high sound quality. In modulation technologies, ARIB choose PSK for satellite broadcasting, Orthogonal Frequency-Division Multiplex (OFDM) + PSK or QAM for terrestrial network and QAM for cable system.

The parameters and specifications vary from standard to standard though there are a lot common technologies used. Table 1 summerizes some of the technical facts regarding all three DTV standards [6].

In Nordic countries (Denmark, Finland, Iceland, Norway and Sweden), a common platform for digital televison, called Nordig, is specified. Nordig is fully compliant with the DVB standard, and it specifies different requirements for different profiles: Basic TV profile, Enhanced profile, Interactive Profile and Internet Access profile. All requirements are specified in Nordig-Unified Requirement Specification [7].

2.1.3 Adoption of DTV

Incented by the benefits brought by DTV, nations around the world are actively planning to replace their existing analog television broadcasting system with digital system. Some countries have even set up a ‘deadline’ to terminate analog broadcasts. For example, Federal Communication Committee (FCC) in USA mandated commercial broadcasters to switch to digital by May 2002 and public broadcasting to be digital by May 2003. By 2006, according to FCC, all stations should give back their analog channels [8]. United Kingdom also set up its deadline for switchover, starting at 2006 and ending by 2010.

However, the determinent driving force of DTV adoption is not government policy, but the marketing acceptance. High receiver prices, insufficient digital TV content and high expense for broadcasters to upgrade systems are obstacles to switch from analog to digital. So far, in US, there are only 172 of approximately 1250 commercial stations operating DTV signals. The Independent Television Commission (ITC) in London also warns that it is still too early to talk about analog swith-off dates [9].

Table 1. A summary of different DTV standards. ARIB (Japan) ATSC (USA) DVB (Europe)

Video format 1080i: 1920 or 1440x1080, 30fps 480i: 720x480, 30fps 480p: 720x480, 60fps 720p: 1280x720, 60fps 1080p: 1920 or 1440x1080, 60fps

1080i: 30fps 1080p: 24 or 30fps 720p: 24, 30 or 60fps 480p: 24, 30 or 60fps 480i: 30fps or other

576i: 25fps 1152i, 1080p: 25fps

Coding MEPG-2 video, MEPG-2 AAC

MPEG-2 video, AC-3 MEPG-2 video, MPEG-2 audio

Multiplexing MEPG-2 transport stream Transmission Satellite Terrestrial Cable Terrestrial Satellite Terrestrial Cable Info Rate (Mbits/s)

52.2 23.4 29.2 19.39 29.2 24.1 38.1

Bandwidth (MHz)

34.5 6 6 6 26~54 8 8

Modulation PSK

OFDM+PSK, QAM

QAM 8-VSB QPSK COFDM QAM



Despite the uncertainties, experts firmly believe the transision from analog television to DTV is inevitable; only the timing is the problem. Most countries have started their digital broadcasting during the last couple of years. Except US and UK, Finland, for instance, officially lauched commercial digital broadcasting on August 27, 2001, being the first to start using Multimedia Home Platform (MHP) at national level. MHP is a standard proposed by DVB to define a generic interface between interactive DTV applications (cf., Chapter 3). China also finalized its plan of switching from analog television to digital in May 2003, and trial system will be started during the end of 2003.

2.2 Digital Set-Top Box

In order to receive DTV signals, a receiver is needed. The receiver can be embedded into the TV set; or it can be a separate device called digital STB. For most customers STB is more economic, because they can reuse the analog TV set to watch DTV programmes – they just need to connect STB with the TV set. Personal Computer (PC) users can also attach a DTV card to their PCs to receive programmes, but average customers are not likely to do so.

The history of digital STB stems from “cable STB,” which was introduced to compensate the deficiencies caused by the differences between the broadcasting and cable signal environments [10]. In digital era, however, the role of a digital STB plays is not only a broadcasting signal-receiving device, but also a computing device and multimedia entertainment device [11].

From the user’s perspective, digital STB is a signal hub for all the information flows involved in the consumer side of DTV environment. The main input of this hub is RF broadcasting signal, and the main output is audio, video, and graphics to be displayed on the TV monitor. Digital STB also takes in any user input and executes the corresponding application as a react, making the user feel he/she is interacting with the television program directly.

2.2.1 STB Functionalities

The main functionalities of a digital STB consist of broadcasting signal processing, access right verification, audio/video output, applications execution, and interaction. These functionalities are discussed in the following section in detail.

• Broadcasting signal processing: The capability to process DTV-broadcasting signal is a fundamental requirement of a digital STB. On the air, the broadcasting signal is in analog form with normal channel bandwidth of 8MHz. The STB needs to select a certain range of Radio Frequencies (RF) in accordance with a channel, which is called tuning. The output of tuning needs to be demodulated - extracting the original signal from carries. In addition, the STB should perform error correction on the demodulated data to protect the integrity of transport stream.

• Access right verification: Access right control is the basis for almost all DTV business models, from Internet service to video on demand. The STB descrambles the transport stream based on Packet Identification Number (PID), which defines the program or service a packet belongs to. In addition, Entitlement Control Messages (ECM) and Entitlement Management Messages (EMM) or other CA data carried in the transported stream is also extracted during demultiplexing and passed to the CA



subsystem for verification. Certain security model is applied then to protect the content from unauthorized viewing.

• Audio and Video output: Audio and video information in MPEG-2 transport stream are separated by demultiplexing. MEPG-2 video signal is encoded with the format of Y/Cr/Cb, which needs to be converted to the format suitable for TV monitor to display, because TV monitor uses different format like PAL [12]. The Y/Cr/Cb signals are converted to analog RGB and then fed to TV video input port and finally displayed in a traditional TV set. The stereo audio signal, however, does not need any format conversion, and it is forwarded to the audio input port right after digital-to-analog conversion.

• Application processing and rendering: Executing both resident application and application-on-the-air is a must-have feature of modern digital STB. After multiplexing, the service data, i.e. application code or data, is extracted from the MPEG-2 transport stream, and some resident program is activated to execute the application downloaded. The STB then renders the result of excuting on the TV set screen along with the video, by means of overlapping or On Screen Display (OSD).

• Interaction: The addition of a dedicated interation channel makes viewing television a two-way interaction, rather than passively watching. With the application running, viewers can navigate and select the content they are interested, using wireless keyboard or remote controller. Viewers’ input can be forwarded to channel owner’s servers through modem, and these servers could analyze users choises and act accordingly.

2.2.2 STB Hardware Architecture

Multimedia PC is often used as a reference when discussing the STB’s hardware architecture. A multimedia PC has a fast CPU, large amount of memory, and a dedicated graphics card to process and render audio and video signal in realtime. While multimedia PC is a versatile computer and can execute a wide range of programs, STB is especially designed for DTV signal processing, with limited amount of applications running on it. Multimedia PC usually has floppy disk and hard disk while low-end STB is most likely not equipped with any external storages. Otherwise, a STB is quite similar to a tranditional multimedia PC in physical hardware configurations.

Generally, the physical components of a STB can be categorized into five subsystems: Front End, Demultiplexor and Descrambler, Decoder, Microcontroller, and Output Encoder. A conceptual view of modern STB architecture is dipicted in figure 2, though the real-life implementation might be different from vendor to vendor. The following section describes each subsystem in detail.

• Front End: Front end is the module to receive and demodulate DTV broadcasting signal. The first component is the Tuner, which tunes to the right frequency range and convert the incoming signal to base-band. The A/D converter then converts the analog signal into digital signal, and the demodulator extracts the modulated bit flow from the base-band. Finally, Forward Error Correction (FEC) component corrects the



transferred data errors that might happen during transmission. As a result, a digital transport bit-stream is recovered, ready for conditional access verification.

• Descrambler and Demultiplexor: As discussed in section 2.2.1, descrambler separates packets from the transport stream based on their PID, and demultiplexor extracts the video, audio, service application and data and feeds them into respective decoders. Access right verification is basically performed in two parts: descryption and descrambling. The coded descrambling control words or keys are first descrypted with the help of microcontroller, and these keys are then utilized to descramble the bit-stream. Figure 3 shows the diagram of CA subsystem and its interaction with

Figure 2. Digital STB architecture [10].

Smart card

Microcontroller

Descrambler

Demultiplexer

Conditional Access

DemultiplexerMemory

Transport Bitstream

Output

data

PID

Figure 3. Diagram of CA subsystem [12].



microcontroller.

There are two different ways to implement the CA system in a STB: embedding it in the STB, or using Common Interface (CI). For embedded implementation, DVB has developed a common descrambling algorithm called SimulCrypt, which allows different STBs with various CA systems to descramble the same content. In this case, the broadcastor, however, has to include scrambling keys for different CAs. CI is also defined by DVB as an open interface, and the descrambling is done in a Personal Computer Memory Card International Association (PCMCIA) card, which is a separated component and could be plugged into STB’s PCMCIA slot. CI allows a STB to use different CA systems, by just changing the PCMCIA card. This approach is called MultiCrypt in DVB standard.

• Decoding Subsystem: After multiplexing, audio, video, and data stream are forwarded into the respective decoding components. The video decoder decompresses the MPEG-2 video stream and recovers the picture frame sequencies from encoded I-, B- and P-frames. The video decoder is also capable of formatting pictures for different TV monitors with different resolutions and different aspect ratio, 16:9 or 4:3. Audio decoder translates the audio stream into sound signal for play out. Audio decoder also extracts the information to synchronize video and audio output. Usually, the audio decoder is implemented along with the video decoder in one chip to simplify its design.

The data stream flowing out from the demultiplexor is forwarded to the data decoder for interpretation. The data stream contains channels information and interactive services, which are encoded in table format. The data decoder interprets this table-formated information and sends it to the set-top microcontroller for further processing.

• Microcontroller: The configuration of the microcontroller in STB resembles that of a personal computer. It has a Center Processing Unit (CPU), memory modules, a graphics processor and a set of I/O ports. CPU initializes and controls various STB hardware components and executes all the interactive TV applications. In addition, it manages the hardware interrupts, which is essential for real-time interaction. Memory modules typically consist of Read Only Memory (ROM), Random Access Memory (RAM) and flash memory. ROM persistly stores data even when the power is off, while the content of it could only be written by flashing. RAM loses its data after power-off, but it’s content could be written and read, and it is cheap. ROM is usually used to store short bootstrap program to boot up the STB, and RAM is used to hold applications. Flash memory is basically one type of ROM, but it provides the possibility for applications to write data. Digital STB usually uses flash memory to store the bootstrap programs, operating system and residential services, which could be easily upgraded by broadcasters.

Graphics processor is dedicated for rendering application information onto the TV screen, along with the TV program. Video memory is also needed to buffer the display, and multiple logical display panels are necessary to support overlapping and blending.



The falling price of hard disk in past few years make this optional component common in modern STBs, and enormous mount of data can be stored permenantly with very low cost – 120 GB harddisk only costs about 130 dollars when this thesis is written.

The output interfaces of digital STB are diversified, and almost all interfaces used in PC could be found in STB. The major interface is Scart, which connects the STB to TV-set. In addition, IEEE1394 could be used to transfer data between STB and multimedia devices with a speed up to 400 Mbps. Universal Serial Bus (USB) and infrared are usually utilized to connect keyboards, joysticks with STB. If there is an interaction channel, modem or Local Area Network (LAN) interface card is needed.

The performance of STB largely depends on that of the microcontroller. Thus, some standards also specify the minimun hardware requirements for STB to complete certain tasks. NorDig, for example, specifies the following memory requirement should be met for Enhanced profile DTV: [7]

- 16 Mbytes RAM

- 4 Mbytes video RAM

- 8 Mbytes flash memory

• Output Encoders: The purpose of audio encoder and video encoder is discussed in section 2.2.1. Audio encoder for stereo sound is actually a D/A converter. Some advanced audio playback systems support digital audio input, thus the D/A converter can be bypassed. Video encoder is far more complex, because video format conversion (Y/Cr/Cb to RGB) is needed. (cf. section 2.2.1)

2.2.3 STB Software Platform

The software architecture of digital STB is quite close to that of personal computer. However, due to the special computing environment, the software platform of a STB needs to fulfil more restrictive requirements:

• Real-timeness: The software system in a STB should be fast and capable of functioning in a real-time environment. The response delay should be very short, because most viewers won’t be patient to wait for longer than ten seconds before something happens [13]. Therefore, a real-time operating system is absolutely needed in STB.

• Operational with limited hardware resources: As a massive consumer device, a STB does not have the processing power of an expensive multimedia PC. Memory is also very limited, and they are hardly upgradable. Hard disk is quite rare in low end products. The limited hardware resource needs STB software to be compact, efficient and extremely reliable.

• Excellent usability: The user group of digital STB covers virtually all the population, no matter what the educational background they have. This requires the User Interface (UI) of STB software to be very intuitive and easy to follow. In addition, the fonts and layout of display should be very clear, because most families watch



television at the living room, and the distance from TV is at least 3 meters, comparing to half a meter from a desktop monitor. Remote control is most likely the only input device for most TV viewers, and interaction between STB and user should be very simple and straightforward.

• Easy to update: Most viewers do not have the knowledge to update the resident software in a STB. As a result, it is required the STB should be able to update its software with minimum involvement from the user. The software should be stored in flash memory or hard disk, and broadcastor initiates the updating automatically.

Figure 4 shows a conceptual view of digital STB software architecture. There are altogether four layers: hardware drives, operating system, middleware and service applications. Following paragraphs provide detailed description of each layer.

• Hardware device drives: Hardware device drives provide vendor specific programs to drive the devices in a STB. A well-designed device drive layer should provide abstract interface for the operating system to communicate and control hardware devices, and this abstract interface should be independent of hardware vendors. By complying with the same abstract interface, STBs from different vendors could support the same operating system. As a result, hardware vendors do not need to develop their own operating system, which lowers the threshold to enter the market.

• Real-time operating system: The operating system in a STB manages all the hardware resources and keeps them accessible to applications through API. Event-driven model is wildly used in STB operating system design to fulfil the requirement of real-timeness. The operating system needs to be loaded when a STB is powered on, and a special piece of software called bootstrap is first executed in this case.

There is not standard set-top real-time operating system, though a shared open system would be beneficial for both vendors and customers. Many broadcastors and

Figure 4. STB software archtecture [10].



consumer electronic manufacturers develop their own operating systems for STB, and currently the existing STB operating systems are: [10]

- PowerTV OS

- VxWorks

- pSoSystem

- Microware’s DAVID OS-9

- Microsoft’s Windows CE

- Embedded Linux

• Middleware and resident applications: The middleware layer abstracts the interaction between application and operating systems. With the help of middleware, set-top applications do not need to care about the details of operating system and hardware configurations. The layer of abstraction significantly reduces the complexity of application development. Virtual machine is an important form of middleware in STB environment. A virtual machine not only abstracts the behavior and function of the hardware, but also defines the common apperance and representation of applications. The most common used virtual machine is Java Virtual Machine (JVM).

As a widely accepted middleware standard in for STB, MHP defines a common API, which provides a platform independent interface between applications and hardwares from different vendors. MHP categorizes set-top applications into three profiles: enhanced broadcasting, interactive broadcasting and Internet access. DVB-J platform, the main component of MHP, consists of a Java virtual machine and a set of television-oriented APIs. In addition, MHP also defines multimedia content formats, DVB Hyper Text Markup Laguage (DVB-HTML) and network protocols, such as Digital Storage Media Command Control (DSM-CC) data/object carousel for broadcast and Internet Protocol (IP) for return channel.

In addition to middleware, resident applications are also integrated at this layer. Resident applications are flashed into the flash memory when the STB is manufactured, and they are upgradable after installed in the DTV network. Navigator, Channel information, and Electronic Program Guide (EPG) are typical resident applications.

• Downloadable service applications: Most interactive services, like Web browsing and gaming, are provided by downloadable applications. These applications are developed based on middleware layer APIs to gain maximum compatibility, and they could be distributed through MPEG-2 transport stream or IP network. The addition of these applications makes the digital STB a versatile home entertainment device.

2.3 DTV Applications and Services

DTV offeres a revolutionary platform for a number of new services and applications, some of which have never been seen in analog television system. For example, because compression



in DTV makes more channels available, a broadcaster could initiate multi-camera angle service – a viewer can select different camera angles when viewing a football match, with one angles transmitted on a separate channel. MPEG-2 standard allows application and service data to be multiplexed and transmitted with audio and video data, which provides an endless source for STB applications.

Given the wide applications and possibilities provided by DTV, it is quite difficult to enumerate all the DTV services. However, we can roughly categorize DTV services into three classes: enhanced information services, enhanced television and interactive services.

2.3.1 Enhanced Information Services

Information services are perhaps the most fundamental form of interaction provided by DTV. The service data is multiplexed into MEPG-2 transport stream, and user can interact with the service only after the content is available in the STB, which is also defined as ‘local interactivity’. Viewers just have the flexibility to scroll and read the content in web-like format, and the data is pushed by the channel owner. Teletext is a typical example of this kind of service in analog televison, and has existed in Europe since 1980’s. In Teletext, text stream is transmitted during the vertical blanking interval of the analog broadcast signal. In

digital era, super Teletext, which could present rich content including text, graphics, sound, and video, will replace tranditional Teletext eventually.

Lots of services could be delivered through super Teletext, for example, news, weather report, financial information and TV advertising. Super Teletext could be implemented as a standalone application or as a feature of generic web-browser.

Another common application in this category is EPG, normally a resident application distributed in STB’s flash memory. EPG provides program information for hundreds of channels, and viewers can navigate these inforamtions through well-organized UI. The functionality of EPG is generally intergrated with that of the TV tunner, and viewer can jump

Figure 5. An EPG application from Gist [14].



to the interested channel by pushing a button in EPG. Intelligent EGP also provides searching and personalizing features, with which TV viewers can search a program and configure the TV to open a program at specific time [15]. Figure 5 shows a commercial EPG program from Gist in action.

2.3.2 Enhanced Television

Enhanced television usually means that traditional televison programming is supported by additional information flows, like multiple video streams and text-based supporting information. STB analyses user’s input from remote controller and presents different program parts accordingly. For example, digital documentary, usually distributed by using DVD, is accessible to massive audience through DTV broadcasting now. ABC’s documentary called Long Way to the Top – Live in Concert has been on shown starting from November 2002, [16] and a survey reveals 81% of the viewers are positive about the service [17]. This program is carried out by distributing the content on 4 separate video streams, and the viewer can switch to the beginning of each stream by pressing a key. Figure 6 shows the main screen of the program.

Interactive sports service could also be implemented by this means. For instance, two or more simultaneous football matches could be broadcasted at the same time, in different video streams, and they can also be displayed on the TV screen side by side. The viewer can then follow multiple games, without missing any important episodes.

2.3.3 Interactive Services

Interactive services require an interaction channel, usually through a Public Switch Telephone Network (PSTN) modem or cable modem. Some services do not need high bandwidth or responsiveness; under some circumstances, even a constantly open channel is not necessary. Viewers can interact with applications by using remote control or wireless keyboard, and their input will then be forwarded into broadcaster’s server.

Figure 6. Main channel of the program Long Way to The Top, user can view different video stream

by select the menu item on the bottom [16].



Sending and receiving email is the first obvious application of this service, because email does not cast restrictive requirement on network connection. In addition, Short Message Service (SMS) and Multimedia Message Service (MMS) could be other potential applications. Chat in Internet is very common now, and TV chat is also possible given the availability of bandwidth. Chating allows viewers to communicate with each other when watching TV programs, exchanging their opinions about TV shows in real time. Figure 7 shows the picture of AOL TV’s Email and chat applications [18].

Gaming on DTV is an exciting service for broadcasters and viewers. Video game industry is growing extraordinarily fast. In 2002, the revenue of video game industry in US exceeded that of movie industry for the first time, and Datamonitor predicts gaming in DTV will bring $2.7 billion by 2006 [19]. Most DTV games currently are quite simple, with 2D graphics and single player mode, and they are distributed to subscribers for free. Future DTV games will finally resemble the games in PC and console, with fully 3D graphics, network connection and multiple-players support. As we all know, 3D video games require intensive computing power, and manufactures are constantly seeking ways to increase the capabilities of STB hardware. In addition, some software companies also proposed methods to promote DTV gaming. For example, a Finnish company called G-cluster has announced their technology to facilitate 3D gaming on DTV, based on cluster computing and broadband network [20]. Figure 8 shows a sketch of their solution.

E-commerce on television probably is another potential service brought by DTV. Dedicated TV shopping channels are quite common on analog era, but viewers can only read the product information broadcasted. In DTV, with the help of an interaction channel, consumers can select the interested product and do the payment. In addition, traditional bank services and transactions could also be done on DTV. However, full-fledged e-commerce on television is too new a service to predict, and some people suggest the service should be cautiously deployed. Currently, there are some pilot e-commerce services existing, like iTV betting applications on Sky. Video on demand (VOD) is a hot topics coming with DTV technology. VOD allows the viewer to watch a TV program or movie at any time he/she wants, without sticking to the time slot predefined by the broadcaster. VOD could be bound with Pay Per View (PPV) service, giving viewers maximum flexibility in selecting programmes. However, full VOD is very

Figure 7. Email (left) and chat (right) service provided by AOL TV [18].



expensive to implement with current technology, thus people come up with the idea of Near Video on Demand (NVOD). In NVOD, movies are started on different channels in a certain time increments; viewers can easily follow the program at the time slot he/she prefers. Another technology should be mentioned is Personal Video Recoder (PVR), which records the program at predefine time, and viewer can replay it whenever he/she wants. PVR could provide localized VOD, and if integrated with EPG, the recording could be even initiated by the channel owner.

More and more innovative applications and services are proposed in DTV, and it is impossible to permutate all of them in this thesis. However, whether a new service will succeed or not depends not only on the novelty of the technology, but also on massive market’s acceptance. To make it more complicated, the adoption of new services is determined by the culture and political environment, as well as the educational background and personal favoriates of the target users. Some services will succeed in one area, but probably will fail in another, thus it is quite difficult to predict the trend of services in future DTV.

Figure 8. G-cluster's DTV gaming solution [20].


Java in Digital Set-top Box 16

3. JAVA IN DIGITAL SET-TOP BOX

3.1 Introduction

Although the architecture and functionality of a digital STB resembles that of a multimedia PC, people prefer to interpret digital STB as an embedded device. The main reason is that a digital STB is specific for DTV entertainment, while a mutlimedia PC is a general computing device, with the capability to execute a wide range of applications. In addition, digital STB generally has simplified design, and manufacturers intend to integrate many related components, like audio and video decoders, into a single chip, to reduce the cost. Finally, the real-time operating system and applications in a digital STB are more compact and require less hardware resources comparing to their couterparts in a multimedia PC.

The above-mentioned characteristics of digital STB certainly bring good news to software developers. For instance, simplified design frees device drive developers from configuring a large scale of hardware varieties, and specific purpose means developers can use small-footprint libraries and Software Development Kits (SDK) to implement applications. However, these advantages could be easily compromised, if the application is going to run in STBs from different vendors with imcompatible APIs. Middleware, as discussed in section 2.2.3, is a straightforward solution for this issue, and JVM is the most adopted middleware in STB industry.

After released by Sun Microsystems in 1995, Java programming language has been one of the most popular Object Oriented (OO) programming language in software development community. Cross-platform is the most attractive property of Java, which means once the Java program is written, it could then run in any platform that supports Java, without even re-compilation. For STB software developers Java technology offers the opportunity to write a program once and depoly it in various STB systems, which will definitely reduce the time to market and development cost.

Belive it or not, the history of Java started from a Set-top TV like project in Sun, 1991, [21] and the initial goal was to create a language, which could shield developers from the complexity and variety of consumer electronic devices. However, Java was widely accepted as a general purpose programming language after its release, because of the booming of Internet, and people needs an elegant OO language to deploy identical solutions to different platforms, like active web contents.

Ironically, Java in its early stage was not suitable for embedded devices, because the capabilities of these devices (e.g CPU speed and memory capacity) were very limited and Internet, the environement that magnifies the benefits of Java, was still in its infant stage. The continuous advancement of semiconductor technology makes embedded devices more and more powerful, and Sun Microsystem has also tried to optimize and segament Java to suit different hardware and platform profiles ranging from enterprise computing environment to smart card platform. For example, Java 2 Enterprise Edition (J2EE) provides enterprise solution; Java 2 Standard Edition (J2SE) is for most common applications in PC environment; and Java 2 Mobile Edition (J2ME) focuses on embedded devices. Thanks to their efforts, nowadays, Java is very commonly seen in embedded systems.



Using Java in embedded systems offers lots of advantages to developers, and the most significant are platform independence, ease of development and reuse, and unprecedent degree of security and stability. Following sections discuss in detail the pros STB software developers gain from Java technology [22].

• Ease of development and reuse: Java programming language is pure OO language, intensively utilizing design patterns and object-oriented methodology. This design philosophy gurantees that Java components are quite easy to develop and reuse. Platform and hardware independence means “write once run anywhere” is possible, in addition, it also means the programmers do not even need a target device to start their developments, which will expidate the development process.

• Native code integration: With the help of Java Native Interface (JNI) technology, Java program code could be easily integrated with native code, like code compiled from C or C++. This presents the opportunity to extend Java’s capability easily. For example, programmers could write time-critical components in C or even Assembly language, and then integrate these components with Java to obtain maximum performance.

• Security and stability: The source code of Java was released publicly by Sun to be scrutinized and investigated by thousands of developers, and secruity issue is of the most important to be considered. Java has its own security policy that requires developers to follow. Moreover, the stability of Java is also testified by millions of users in the Internet.

• Network connection: Network connection is another issue in Java creators’ mind, and Java has powerful capability to handle network connection between embedded devices, which is very helpful for DTV programmers, because they can be relieved from the complexity of network connection coding. In addition, compiled Java programs could be distributed by the network and loaded by the target device automatically, and this is an ideal solution for delivering/upgrading DTV applications.

• Performance: In its early stage, the performance of Java was most critized, and Java was not the preferable language for high-performance applications. However, this is no longer an issue when low-cost powerful chips are availabe. In fact, program larger than 500KB could be more compact when distributed in Java byte-code. With Just-in-Time (JIT) compilation techniques, program can approach native machine language execution speed.

3.2 Java Architecture

To describe the Java architecture clearly, two environments should be discussed first – Java compile-time environment and Java Run-Time environment (JRE). The development work is done entirely in Java compile-time environment, in which the Java source files are compiled into byte-code Java class files by the compiler. These compiled class files are distributed locally or via network to a target device’s JRE. After byte-code is loaded and verified in JRE, the JVM will interpret them, link them with system classes, and execute them with the access to hardware resources.



Figure 9 depicts the diagram of Java architecture. Normally, Java architecture could be divided into four distinct, but interrelated fields [23]: Java programming language, Java API, JVM, and Java class formats. Restrictively speaking, Java programming language and Java class formats are not physically existed components in Java environment, and thus they are not discussed further in following sections.

3.2.1 Java Virtual Machine

JVM is the key component to implement the platform independence mechanism of Java technology. In addition, JVM also provides the functionality of security and network connection. The main task of JVM is to load and verify the byte-code classes and execute them. Figure 9 shows that there are four components comprising JVM: Byte-Code Loader, Byte-Code Verifier, Interpreter, and Runtime Engine.

• Byte-code Loader: Byte-code loader takes the responsibility of loading Java classes into the virtual machine, and it also determines when and how to do this. There are two types of byte-code loaders: the premodial class loader and user-defined class loaders. Premodial class loader, as an intrinsic part of JVM implementation, bootstraps the Java environment and loads API classes. Premodial class loader is typically written in C, and it usually loads trusted classes from local disks because it uses native system’s file access capabilities to open and read class files. User-defined class loaders load classes in customized ways when classes are needed by the virtual machine. There could be multiple user-defined class loaders coexisting in a JVM, each of them defines one way to load classes, for example, applet class loader defines that JVM loads classes via Hyper Text Transfer Prototol (HTTP), and Remote Method Invocation (RMI) class loaders can only load classes from the Unified Resource Location (URL) specified by Java's rmi.server.codebase property.

• Byte-code Verifier: The byte-code verifier of Java machines ensures that loaded classes have proper internal structure. The verifier does the checking in two phases: internal checking and symbolic references verification. In the first phase, JVM checks the class file format and internal consistancy, making sure the loaded class has the correct structure and complies to the constrians specified by Java programming

Figure 9. A conceptual diagram of Java architecture [22].



language. In addition, the JVM also verifies that the stream of bytecode is safe to execute by checking the operand stack. In phase two, JVM verifies the symbolic references, character strings that give the name and possibly other information about the reference item. This phase of verification is done when a class is dynamicall loaded, and binary compatibility is also checked here.

• Interpreter and Runtime engine: Sun’s Java specification does not specify how the Java virtual machine should be implemented, and there are lots of JVMs implemented in hardware and/or software. A pure software implementation of JVM is also called Java interpreter. However, interpreter in this section only means the component that interprets Java bytecode. Interpreter and Runtime engine are fully connected, and result of interpreter is forwarded to the runtime engine, where the system resources are called to execute the bytecode.

It is worthwhile to mention that an optional component, called JIT compiler, could be inserted between the Interpreter and System hardware. JIT compiler is a utility to improve the performance of Java applications, by compiling the Java bytecodes into native hardware specific codes before executing.

3.2.2 Java API

Java API is a set of runtime libraries to provide an unfied way for Java applications to access system resources. The developers of Java programs assume Java API classes are available in target system during development time, and these API classes should be installed along with the JVM. JVM loads Java API classes that are refered to by a Java program, and these classes are combined together to generate an executable Java application.

Java API classes are implemented specifically for different host platforms, and they provide an identical interface for Java applications to utilize system resources. API classes call host systems’ native methods to access the operating system services and hardware resources, like disk read and write, network connecting and timer. Consequently, Java APIs for different platforms have different implementations, because native methods are different from platform to platform. Platform independence is guaranteed by the unified interface that Java API presented to applications. In this context, Java API classes could be understood as an adapter or filter for Java byte-code flows – the input of which is Java byte-code, and the output is control flow for system resources.

Design pattern and OO design methodology are intensively used in Java API, to achive

Figure 10. Java API architecture [22].



maximum expandibility and reusability. For example, JOptionPane is a façade to create different types of dialog and display them, and the mouse and keyboard listener, which Java applications uses to trace the input event from users, are canonical examples for observer pattern. Use of OO methodology makes Java API easy to study and a programmer with OO knowledge can get familiar with Java API very quickly. In addition, Java program developed with OO philosophy in mind takes less effort to maintain and expand than non-OO programs.

Java API classes are distributed in packages, and there are over 50 Java packages coming with Sun J2SE SDK. Beside, a number of Java API is distributed by third-party developers, to enrich the functionality of Java SDK. JNI plays a key role in implementing new Java APIs, by providing a ‘backdoor’ for Java programs to access the native methods. The research work of this thesis depends heavily on JNI technology, and a detailed discussion of JNI is in chapter 5.

3.3 Java in Digital STB

Platform indepedence and intrinsic networking support are key motivations for STB industry to adopt Java technology. STB manufacturers have been very active to provide Java support in their products because they need an open-layer on which STB applications base on, and great efforts are taken to create a generic Java API for digital STB. Currently, there are three widely accepted solutions for set-bop box Java: Sun’s Java TV API, Digital Audio Video Concil (DAVIC) Java centric solution, and DVB-MHP. It needs to mention that they are not separate solutions – DAVIC’s solution contains some Java TV API packages, and DVB-MHP contains some features from Java TV API and DAVIC solution.

3.3.1 Sun’s Java TV API

As the inventor of Java technology, Sun Microsystem plays a central role in providing Java support for digital STB. By working closely with major players in STB industry, Sun released the first version of Java TV API in 1998. From then, programmers started to be able to develop Java TV applications with the help of TV API as well as system API in Java SDK.

Like other APIs, Java TV API abstracts the functionality exposed by the lower-level libraries to control the hardware resources of the Digital STB. A downscaled JVM, PersonalJava, is used as the virtual machine; PersonalJava is typically suitable for devices with constrained hardware resources (especially in term of CPU speed and memory capacity). Major features of Java TV API include: [23]

• Service and service information access: Java TV API defines a collection of content for presentation in STB as a service, which is handled as a unit in API context. SI describes the layout and content of audio&video or data stream, and Java API uses Locator object to refer to SI elements. Java TV API abstracts various service information protocols, thus applications do not need to care about the actual SI protocol used in DTV system.

Based on the application’s needs, different views of SI could be provided by service information database, and each SI view is described in Java TV API as a package. There are altogether four SI packages defined: service package in javax.tv.service, navigation package in javax.tv.service.navigation, guide package in



javax.tv.service.guide and transport package in javax.tv.service.transport. Detail discussion of each package can be seen from Java TV API whitepaper.

• Broadcast pipeline control: In Java TV API, the broadcast pipeline of a STB is represented by Java Media Framework (JMF). JMF defines a comprehensive set of controls for handling video and audio playback, and it also offers the possibility to manipulate some aspect of the media. In addtion, JMF mounts tuner-multiplexor-conditional access subsystem as it audio&video stream source.

• Broadcast data access: Data multiplexed in MPEG-2 transport stream is handled by Java TV API’s data access packages. The API supports data carried in three formats: broadcast file systems, IP datagrams and streaming data. Java TV API supports both DSM-CC object carousel and data carousel, and the API documentation also suggests some ways to alleviate the latency problem caused by these methods. IP datagrams transmitted in the broadcast stream is handled by the java.net package from basic Java API, which means both unicast datagram and multicast datagram could be processed. When there is generic streaming data, JMF package javax.media.protocol could be used to handle it.

• Application lifecycle management: Application based on Java TV API is called Xlet, and an application lifecycle model is defined to control the state change of Xlet, as well as its interaction with the running environment. The application manager, a resident application that controls the lifecycle of Xlets, is required. The implementation method of an application manager is not defined in API, while its requirements are specified. An Xlet has four states: loaded, paused, destroyed and active, and the transition event between these states are define in javax.tv.xlet package. The following picture shows the trasactions between different states.

Sun’s Java TV API provides a unified interface layer for STB applications, which definitely benefits software vendors for DTV services. Nevertheless, a pitfall of Java TV API is that the interfaces are defined in a very high level, due to the fact that Sun had the pressure from various DTV standards organizations, and STB vendor has to extend the API to support features specific to broadcast environments. Another problem is that Java TV API does not contain a TV-centric UI model, and Java Abstract Window ToolKit (AWT) has to be used for user interface features. However, Java AWT is inherently designed for PC/workstation screen, and user interaction heavily relies on mouse clicking, which is not practical for TV interaction. As a result, a TV-friendly UI subsystem is needed.

Figure 11. Xlet state machine [23].



3.3.2 DAVIC Java Centric Solution

Based on Geneva, DAVIC is a non profit-making consortium founded in 1994, with the aim to “promote the success of interactive digital audio-visual applications and services by promulgating specifications of open interfaces and protocols” [24]. There are five series specifications proposed by DAVIC, specifying virtually all fields in DTV context, ranging from physical interfaces to DTV intranet protocal.

DAVIC has also done a lot of work on APIs to enable broadcast of interactive TV applications. Initially, DAVIC used MHEG-5 as a format to transmit page-oriented contents, and a Java centric solution was proposed later. DAVIC Java centric solution utilizes and extends Java TV API packages and some other Java packages. For instance, java.awt is used as user interface package with extension for transparency, and JMF is used for streamed audio-video control.

Besides Java TV APIs, DAVIC also specifies a large set of APIs specific to digital TV broadcasting. Unlike Java TV API, DAVIC specifically designed some APIs for DVB standard. Two types of APIs are proposed: common Java APIs and supporting APIs, and both are discussed briefly in the following sections.

• Digital TV APIs: Digital TV APIs consists of APIs for DVB SI, MPEG-2 section filtering, tuning, and conditional access. SI API provides a high level interface for accessing the DVB SI and MPEG-2 Program Specific Information (PSI). MPEG-2 section filtering API helps applications to to access data held in MPEG-2 private sections, like application specific soft real time data. It also provides different ways to filter information flow. The name of tuning API suggests this API could be used to tune to broadcast transport streams, which abstracts the physical transport medium for high-level applications. Finally, CA API provides applications a basic interface to the CA system.

• Supporting APIs: Generally, supporting APIs are not standalone APIs -- they could only be used by other APIs. This type of API contains basic MPEG API, reference content API and resource notification API. The classes in basic MPEG API are used by other APIs or applications as handles for transport stream and services. DAVIC uses Locator object to reference content, and the Locator object could be constructed using URL strings. The method for Locator object parsing and manipulation is defined in referencing content API. Finally, resource notification API is used for resource availability notification. For example, when a resource is removed in the data source, this API will notify the class who initiates the request, that resource is not available.

3.3.3 DVB-MHP

DVB-MHP workforce was started by DVB group in 1997, with the aim to standarise the key elements in digital STB environment. MHP defines a generic interface between STB applications and hardware, to decouple “different providers’ application from the specific hardware and software details of different MHP terminal implementations” [25].



Based on JVB-J standard, MHP is a convergence from various solutions. Figure 12 shows the diagram for DVB-MHP architecture [26]. MHP includes open standard APIs like Java TV, DAVIC, and the UI part of Home Audio Video interoperability (HAVi) standard. HAVi is a home networking standard to facilitate interoperability between digital audio and video devices, and it provides an excellent user interface API specific to TV, including basic widgets like button and text field, and high-level concepts like containers. MHP’s Java virtual machine is a subset of personalJava 1.2, and some APIs are modified to suit the requirement of DVB broadcasting.

DVB-MHP also specifies various content formats that should be supported in STB platform. These formats include PNG, JPEG, MPEG, subtitles and fonts. In addition to Java-based DVB-J application format, MHP also defines another optional application type named DVB-HTML, which is based on eXtensible Markup Language (XML). With DVB-HTML, content providers could author the XML document once and it will be rendered correctly in different targeting devices like Internet, TV, Personal Data Assistant (PDA), and mobiles. A DVB-HTML application can access to TV resources through MHP Java APIs and it can also embed Xlet applications [27].

Mandatory transport protocols are also included in MHP. These protocols are network-independent in order to make MHP units communicate through different network types. For broadcast interactive services, MHP supports DSM-CC object carousel and data carousel protocols; for interaction channel, IP is supported.

MHP has three profiles to cope with a large variety of hardware configurations [28]. The first profile is enhanced broadcasting, and applications in this profile require limited interactivity via telephone/cable modem return channel. Profile 2, interactive TV, also uses a similar return path as profile 1, but requires stronger support in software platform to perform better interaction. Profile 3, Internet profile, targets at wide band connection and real-time interaction, and it also requires applications to support downloading contents from Internet directly. Profile 1 and profile 2 are specified in MHP 1.0 and 1.1, while profile 3 is only speficied in MHP 1.1.

DVB-MHP also suggests to STB vendors the hardware requirements for each profile. Profile 1 and profile 2 need a CPU with 80-130Mhz, 4MB ROM, 8-16MB RAM and optional hard disk; profile 3 requires 150-200Mhz CPU, 32-128MB ROM, 16-32MB RAM and 20-40GB

MHP Applications

ApplicationManager

Transportprotocols

Sun JavaTV HAVi HAVIC DVB

specific API

Java Virtual Machine

Operating System

Figure 12. MHP architecture.



hard disk [29]. Some countries have started to support MHP in their broadcasting system, for example, Finland lauched its DVB-MHP enabled DTV service in August 2001, and Australia has selected MHP as it de facto API.

Figure 13. DVB-MHP 1.1 profiles [28].


3D Graphics in Digital Set-top Box 25

4. 3D GRAPHICS IN DIGITAL STB

4.1 Introduction

3D computer graphics (CG) is the technology to generate and display three-dimensional objects and scenes in two-dimentational screen space. The history of 3D computer graphics could be traced back to 1960’s, and for 30 years, it had been only accessible for researchers and commercial users because of the high expense of graphics systems. The age of PC 3D graphics started in 1995, when 3Dfx released the first affordable PC graphics card that is able to perform some 3D operations in hardware. Since then, great effort has been made to improve the performance of PC 3D graphics cards. Products nowadays are able to process very complex geometry with trillions of polygons in real time, and the image quality is close to that of a movie [30]. Video game industry immediately benefits from this advancement -- thousands of 3D games are produced every year, and distributed to millions of game players with PCs or game consoles.

The success of 3D graphics in PC and game industry inspires digital STB manufacturers. After all, it is a straightforward idea to integrate 3D graphics capability in STBs, given the architectural similarity shared by a STB and a PC. In addition, researchers believe a TV set should be a better entertainment platform than a PC, because TV screen is larger than PC monitor, and TV also has greater coverage and popularity. However, 3D graphics applications require high hardware capability with fast CPU, more memory, and a dedicated 3D graphics card. As a result, currently, 3D capability could only been seen in high-end digital STB.

This chapter will first discuss the 3D processing pipeline, to explain why 3D computer graphics is so computational demanding. Some major PC 3D APIs will be investigated. The last section studies 3D applications and services in digital STB context, and conducts a feasibility study of 3D graphics in digital STB according to current hardware capabilities.

4.2 An Overview of 3D Computer Graphics

Before the 3D content could be displayed on a computer screen, it has to be processed in two stages: authoring and rendering. Authoring needs creativity and intensive manual work. In this stage, artists model the 3D objects in computer, put them into a scene and define how to animate them if necessary. Rendering displays the 3D object into 2D screen, and the content needs to pass through the 3D rendering pipeline before it is displayed. In practice, these two stages are usually intermingled. For instance, when producing 3D content, an artist might render the content after he/she has modified the 3D model to see the result, and switch to authoring mode again – most 3D authoring tools support this feature. In the context of STB environment, users are only viewing the content, and it is not possible to edit the content in STB. Thus, this chapter only focuses on 3D graphics rendering.

4.2.1 3D Rendering Pipeline

3D rendering pipeline is a chain of operations that convert a 3D object to its 2D representation on a computer screen. These operations require enormous amount of floating-point matrix calculation that are carried on each vertex or each pixel. To understand the rendering



pipeline, a basic knowledge of vector calculus is needed, and readers can get familiar with this in a college mathematic book

Before diving into the pipeline, we need to first discuss how 3D objects are modeled and represented in computer recognizable format. In 3D graphics, the object shape is approximated with a hull closed by flat polygons – increasing the amount of polygons makes the hull a better approximation, but requires more computing resources. The location and shape of a polygon are confined by its vertices. To represent an 3D object, the coordinates (x, y, z) of all vertices, as well as each polygon’s topology information, a list of vertices belong to this polygon, need to be stored. In addtion, other information could also be stored to speed up the pipeline, like polygon normals.

The pipeline starts when it is fed with the digital representation of 3D object. To make it simple and easy to follow, we divide the whole pipeline into two phases: vertex-based geometric processing and pixel-based shading and rasterization. Following sections describe the operations carried out in each phases.

• Vertex-based processing: The major operations in this phase are four transforms between five coordinate spaces. 3D objects are initially defined in its own space, called model space, because it is more convinient to model an object when selecting some characteristic point of the object as the space origin. A scene usually contains multiple objects, as well as cameras and lights, and those entities are defines in their own model spaces. Thus a unified space, called world space, is needed for further processing, and coordinates in model space are converted into world space by modeling transform.

Camera in 3D graphics resembles a human eye, and view transform converts the world coordinates into view space. View frustum defines the volume in which objects could be seen by the camera, and vertices outside of the volume are clipped. Other operations in this stage are back-face culling, to discard the triangles facing away from the camera, and vertex lighting, to apply light equations to each vertex. Projection transform projects a 3D object into a 2D plane, in the way how a camera takes pictures. Finally, screen transform finds the screen coordinates the vertices in projection space.

Figure 14. 3D rendering pipeline.



• Pixel-based processing: In this phase, the pipeline is operating at the pixel level. In 3D graphics, most parameters, like color, lighting, and normal, are defined in the vertex level, and interpolation is needed to obtain these parameters for each pixel. Rasterization determines the screen area a triangle maps to, and it also calculates the degree of transparency of a pixel by alpha test. Depth test measures if a point is hidden. Shading assigns color value to each pixel according to the lighting calculation and color information from the model. Texturing brings more details to the model by maping a picture to the object surface.

It should be noted that the description of 3D pipeline in this thesis is far from complete and detailed – the purpose is just to present readers a general picture about how 3D graphcis works. Most operations, like transforming and lighting, require dozens of floating point multiplication on each vertex. In real applications, it is quite common that a 3D scene may contain over half a million vertices, and these vertices have to be process 20 to 30 times per second. For example, games require the frame rate to be over 20 Frames Per Second (FPS). Pixel processing consumes even more computing power, a screen with the resolution 1024x768 means there are 1 million pixels need to be rendered in 1/20-1/30 second. All of these make 3D graphics very computation demanding, and it might be a luxury just for high end STBs.

4.2.2 3D APIs

Again, a standard software API is crucial to keep the operatability of 3D applications, and 3D libraries are required to encapsulate all the complex operations inside 3D pipeline for programmers. These libaries need to be highly optimized for hardware platform to make the most of computing resources. Usually, a pure software implementation is not practical for real time 3D graphics, even with the fastest CPU in the market, and a dedicated 3D graphics card is needed, which could perform a lot of 3D specific floating operations in hardware. As a result, all 3D libraries should provide a transparent layer between 3D applications and low-level hardware drivers.

In PC 3D industry, OpenGL and Direct3D are the most used 3D APIs. OpenGL is more open than Direct3D, which is owned by Microsoft and only works in Windows operating system. The following section describes both libraries in detail. In addition, Java3D as Sun Microsystem’s 3D graphics solution in Java platform is also discussed.

• OpenGL: OpenGL API was introduced by Silicon Graphics Incorporation in 1992, to provide high performance 2D and 3D graphics features for application developers. The specification of OpenGL is guided by the OpenGL Architecture Review Group, with the purpose to establish a “truly open, vendor neutral, multiplatform graphics standard” [31]. Nowadays, OpenGL is available in a wide range of operating systems, including Windows, Mac, Linux, and Unix. Due to its popularity in 3D graphics world, virtually every 3D hardware manufactures have optimized their products for OpenGL operations.

OpenGL only concerns with rendering into a framebuffer, and a minimum requirement of OpenGL is the existance of an addressable framebuffer. Basically, OpenGL is a state machine that controls the state of a framebuffer by intepreting the commands



from upper-layer applications. OpenGL commands are in simple and intuitive syntax, which makes it easy for programmers to get familiar with.

It should be mentioned that OpenGL is just a set of API, and the actual implementation is up to the vendors. When there is no hardware acceleration, OpenGL should be implemented purely in software, which is certainly very slow. Platform vendors can tailor a particular OpenGL implementation to meet their requirement in terms of cost and performance, and individual OpenGL calls could be customized to be executing in dedicated hardware or in system CPU. In addition, OpenGL extensions offer a mechanism for 3D applications to access to hardware features that is not standard in a particular OpenGL impelmentation.

• Direct3D: As a proprietary API owned by Microsoft, Direct3D only works in Microsoft Windows platforms. Direct3D is a subset API of Microsoft’s PC multimedia APIs, DirectX. Unlike OpenGL, the implementation of Direct3D is not customizable by third party developers after it is released by Microsoft. Direct3D is widely accepted by game developers because of the high coverage of Windows platform. However, if the application needs to be cross-platform, OpenGL might the only choise.

Microsoft has taken great effort to improve the performance of Direct3D, by seamlessly integrating the library with Windows operating system and actively helping hardware vendors to develop optimized drivers. Direct3D provides an elegant way for applications to leverage the features of graphics hardware – application could issues a request, by assuming a certain feature is available in the hardware, and when it is not available, Direct3D could fall back to the software implementation of this feature.

Microsoft also supports Direct3D in its embedded operating system Windows CE. For instance, Sega Dreamcast is a game console based on Windows CE, and Direct3D applications could be working on it. This gives an advantage to embedded system developers – if they use Windows CE, they could use Direct3D too.

• Java3D: Java3D API is Sun’s solution to 3D graphics in Java platform. Bacause Java technology is inherently platform independent; applications based on Java3D are independent of low layer operating system too. Unlike OpenGL or Direct3D, Java3D defines a set of high level API to facilitate 3D scene construction and managing, which is usually done by a dedicated 3D engine in traditional PC 3D graphics industry.

Java3D API does not directly implementes the 3D rendering pipeline, rather, it depends on low level rendering API to do the rendering, namely OpenGL or Direct3D. For windows platform, developers can configure Java3D API to use either one, however, for Unix/Linux platform, OpenGL is the only option.

4.3 3D Graphics in Digital STB

So far, the discussion in this chapter has focused mainly on 3D graphics in PC environment. 3D application in digital STB is a relatively new topic due to the fact that computational capability of STB is limited and there are not solid business cases. Nevertheless, it could be



foreseen that 3D graphics will prevail in digital STB in the coming years when powerful hardware is introduced into mass market and customers start to get interested on 3D contents.

4.3.1 3D Applications in Digital STB

Generally, any 3D applications in PC could be deployed in a STB. However, it has to be mentioned that 3D application in STB has different requirements than its counterparts in PC. First, screen resolution in a TV monitor is usually lower than that of a PC display, which means 3D graphics in STB needs fewer pixels. Second, viewers normally watch TV 3-5 meters away, while a PC user watches the display half a meter away. Long viewing distance requires less detail of content. As a result, a 3D model in STB could be coarser than a model in PC, and thus less vertices and triangles are needed. Last, STB users usually do not have a wireless keyboard, and a remote control might be the only input device. This requires the interaction between the user and 3D application to be minimum and effective.

Though there are no 3D applications and business cases existing in current STB industry, some applications could still be predicted. 3D video games and 3D enriched content are the most obivious ones. Detailed discussion is included in the following subsections.

• 3D video games: During the past years, probably the most successful business in PC industry has been 3D video games. It was reported that the revenue of video games in US exceeded that of movie industry in 2002, and gaming in DTV will bring 2.7 billion USD by 2006. [19]

3D video games are developed for different hardware platforms, including PCs, dedicated game consoles, e.g XBOX, PS2, GameCube, and mobile phones. The games, including applications and data files, are delivered to the target machine by CD, DVD, or by online downloading. Once a 3D game is installed in the target machine, it runs locally – doing the entire 3D computing in the local machine.

This paradigm requires three prerequisites: an effective distribution medium that is able to deliver massive amount of data quickly, enough local storage and enough 3D computation capability in the local machine. It is usually not practical to assume that a current digital STB has CD/DVD drive or broadband network connection because these features will increase the cost of a STB drastically. Fortunately, the unique content distribution model of DTV could be utilized for delivering game content. For example, a broadcaster can reserve one channel dedicated for gaming, by distributing serveral games circularly, in an interval of 15-20 minutes. Viewers can then download the interesting games, save it in the local machine and play it afterwards.

Perhaps the only way to meet the requirement of local storage in a STB is to install a hard disk, and this will inevitably increase the cost. 3D video games need massive storage space to hold the executable code, geometric data, textures, animation scripts and video clips. It is not unusual that a commercial 3D game takes over 1GB of hard disk space. Good news is the price of large volume hard disk keeps droping, and 100GB hard disk is accessible to average consumers currently. In addition, hard disk is not the exclusive investment for 3D games, and lots of other STB applications will take the benefit of a hard disk. For example, hard disk is an indespensible component for Personal Video Recoder (PVR) application.



Limited computing capability is another obstacle in deploying 3D games in STB platforms. As discussed in section 4.2, 3D application is particularly computational demanding, and usually a dedicated 3D graphics card is needed to achieve real-time 3D game effects. Fortunately, STB manufacturers intend to follow the path of PC advancement by continously boosting the STB CPU speed and increasing memory capacity, and some vendors have even tried to integrate 3D graphics chips in their STB design.

Some DTV software vendors are also very active in finding innovative solutions for deploying 3D games in STB platforms. For instance, Lime Studios and G-Cluster proposed a similar solution [32]. The basic idea of their solution is relay 3D computing to the remote server, while the STB is just used to accept user’s input and render the MPEG game content rendered by game servers. Game servers are very

powerful personal computers with 3D hardware acceleration, and they accept the user input from STB, do 3D calculation, and render the scene into MPEG stream. The render result is then forwarded to the target STB and the user can see it immediately. Restrictively speaking, the STB does not get involved in 3D computation in their solution, rather, it is just used as an input and output device. High-speed servers and broadband network are also needed, which will increase the operator’s investment. To make things worse, a player needs one separate session/process running in the game server, and increasing the number of players will inevitably makes the system more complex and harder to manage. In addition, local computing power of a STB, though limited, is totally wasted in this solution. Finally, an existing 3D game in PC or game console needs to be modified before deploying to STB platform. All these facts indicate this is just an intermediate solution, and an ultimate solution should been found.

• 3D enriched content: 3D enriched content means a 3D model is delivered along with other media format, e.g., text, still image and motion picture. In presentation software, the 3D model is embedded and presented with other media formats. Viewer can interact with the model through the remote control or wireless keyboard.

3D enriched content in teletext would be a promising service for 3D graphics. A teletext application that is able to process embedded 3D content could present the text and 3D model simultenously. For example, in teletext weather service, a 3D model of the globe could be displayed along with the text weather information. The globe will rotate to the corresponding city when its weather info is reported.

When embedded with video stream, 3D content could provide a lot of additional information about the program under broadcasting. For instance, in a football game,

Figure 15. Lime Studios 3D game solution for STB [32].



usually the viewer can only watch the game from one camera angle, and he would like to get a bird view of the whole field. The 3D model of the field and players could be embedded in the video stream and presented in a small window of the TV screen.

People will argue there is no need for 3D graphics in the above-mentioned applications – another video stream could do the job. However, another video stream means an additional channel is needed. Moreover, a viewer cannot interact with the content broadcasted with a video stream. For example, if the viewer wants to rotate the football field in a video stream, he cannot do it. But a 3D model could be fully manipulated by the viewer, like zooming, rotating, and animating.

3D Virtual Reality Markup Language (VRML) plugin for STB browsers is another potential applications of 3D graphics. There are large amount of 3D models distributed in 3D VRML formats via the web, and a VRML plugin makes these models accessible to STB browser users.

• 3D interactive applications: 3D interactive applications provide unprecedent interactivity for viewers to control and manipulate the 3D models. This type of application could be a mixture of multiple media formats, including 3D models and animations, video clips, text, sounds, and pictures. The basic senario is a user controls the 3D model, and the relative information in other formats will be played. TV advertisement could be a good business case for this. For example, a TV advertisement about the latest car will start with a 3D model of the car. Viewer can rotate the car and select different components; a text description of the feature of the selected component will be displayed.

3D wallpaper could be another interesting application. With the advancement of display technology, TV monitors can show shaper image with high resolutions and free from flickers. Nowadays, plasma monitors could be less than 10 centimeter thick. This makes TV monitors a good candidate for decoration. 3D scenes and animations could be displayed when nobody is viewing TV. For example, an imaginary deep-sea scenary could be displayed with animated fishes and sea grasses. Different wallpaper profiles could be downloaded from the broadcaster’s servers.

4.3.2 Feasibility Study of 3D Graphics in STB

As mentioned in section 4.3.1, deploying 3D graphics in STBs needs to meet three requirements: an effective distribution channel, sufficient local storage space, and enough 3D computing power in target device. Distribution is not a problem for DTV, because DTV provides a unique channel for distributing large amount of data quickly. The latter two requirements are concerned with digital STBs, and this section will focus on the hardware configuration of current digital STBs to study the feasibility of deploying 3D graphics in STB platforms.

It is very difficult to do a complete survey of the hardware configurations of all the STB in market in a short period of time. In this section, three products from different vendors are selected as subjects to study the hardware capabilities of their solutions. The prices of these products range from USD200 to USD 500, and they represent the main stream products in current or near-future STB market. Among these products, Nokia Medial Terminal [33] is a



complete product, while the other two, ATI XILLEON 220 [34] and VIA VT6010 Mini-iTX [35] are reference frameworks proposed by ATI and VIA respectively, and STB manufacturer has to customize their product based on these reference frameworks. In particular, the main board, CPU, graphics chip and decoders are designated by ATI or VIA, and OEM vendors only need to customize the tuner, CA subsystem, memory, and hard disk storage. When study the hardware configurations of these products, the author only investigates the CPU speed, memory, hard disk, and 3D graphics capacity of these solutions, because they are crucial factors to decide if a system is capable of doing 3D real-time graphics.

Table 2 shows the crucial hardware configuration data for each product. The CPU speed of these products exceeds 300 Mhz, and they all possess 3D hardware acceleration capabilities. Nokia Media Master, as a final product, has 64 MB system memory and 20GB hard disk. ATI and VIA solution, however, do not specify the exact amount of memory and hard disk, but provides the interface for expanding system memory and integrating hard disks. OEM vendors could install memory and hard disk according to their customer’s requirements.

The hardware configuration of current STBs is close to that of personal computer in 1998-1999 mainstream markets, when 500Mhz CPU, 64 MB memory and limited 3D graphics cards were popular. Of course, this configuration is not able to run the latest 3D applications that are designed for systems with 2GHz+ CPU, 512MB meomry and USD 500 graphics card. However, we are targeting set-top platforms, and most users are not professional gamers. 3D games developed 4-5 years ago (Quake II, for example) could be good candidates for current STBs.

The conclusion of STB hardware capability study is that lightweight 3D graphics is feasible with current STB hardware configurations. It is also believed that STB will become more and more powerful in 3D processing and 3D applications will prevail in digital STB in forseenable future.

Table 2. Hardware configuration of Nokia, ATI and VIA solutions. CPU Memory HDD 3D

Nokia Media Terminal

366 Mhz Intel Celeron

64MB system memory, 4MB SDRM for video

20GB Hardware acceleration (per-pixel alpha blending)

ATI XILLEON 220 300 Mhz processor

Memory expandable Intergrated disk interface

Hardware 3D acceleration

VIA VT6010 Mini-iTX

C3 processor 500MHz

Two DIMM socket, expandable, 32 MB rom

2 IDE connectors

Integrated AGP 2x with 3D acceleration


Framework Design and Implementation 33

5. FRAMEWORK DESIGN AND IMPLEMENTATION

5.1 Introduction

Developing and deploying 3D applications in STBs is quite challenging, because 3D graphics is computational demanding, but STBs targeting for mass market have very limited computing resources. Fortunately, with current high-end STBs, lightweight 3D graphics is feasible, and it is absolutely certain that STBs will become more and more powerful in the future. More 3D applications will be deployed in STB platform, and thus a common 3D API layer is needed to facilitate the development of applications.

This chapter mainly focuses on the design and implementation of a Java 3D framework, called TVGL, based on Ubik, the STB platform proposed by Telecommunication Software and Multimedia laboratory, at Helsinki University of Technology. An overview of this STB platform, including the hardware and software, is discussed first. The second part is focused on the architecture design of the 3D framework. Some key issues, like 3D rendering API, JNI, and interface definitions, are discussed in detail in this section. Finally, the last section will evaluate the whole framework and give suggestions to improve it in the future.

5.2 An Overview of the Reference STB Platform

Ubik from Telecommunication and Multimedia lab in HUT presents a versatile environment for STB applications, and TVGL is also designed in mind to be able to work on it. The hardware configuration of this platform includes a board PC with 233 MHz K6-2 processor, 32 MB RAM memory and 64 MB ROM memory. However, this configuration was proposed in 2001, which is obsoleted and not capable of running complex DTV services. As a result, currently Ubik is developed and running on a Linux PC with more capable hardwares.

Linux is used as the operating system for Ubik, due to the fact that Linux is open source, and easy to be ported into new hardware platforms. In addition, a lot of work has been done to reshape Linux, making it suitable for embedded platforms like mobile phones and digital STBs. Though not a real-time operating system, Linux could be equipped with real-time capabilities with the help of add-ons. [26]

The standard Linux for PC was reconfigured, and some components were removed to satisfy the strigent memory requirements. DVB standard has its own specification for user interfaces, and X-window system, which is designed exclusively for PC environment, is not applicable. Consequently, X-window subsystem was removed and a new Havi-compatible GUI subsystem was used instead, which renders the GUI contols into the video framebuffer directly [36].

To archieve maximum compatability with major DTV standards, Java was selected as the main programming language of applications. Kaffe from Transvirtual Technologies Inc. is applied as the Java virtual machine in this environment. The implementation of Kaffe is independent of X-Windows system, and it is capble of working with framebuffer device with Single DirectMedia Layer (SDL) support, which makes Kaffe an ideal solution to JVM in embedded Linux platforms.



I n t e r o p e r a b l e A p p l i c a t i o n

C o r e J a v a A P IX M L P a r s e rJ M F A P If t v P a c k a g e

N a v i g a t o rD i g i t a l T e l e t e x t

A p p l i c a t i o n M a n a g e r

K a f f e J V M

L i n u x O p e r a t i n g S y s t e m

Figure 16. Software architecture of HUT STB reference implementation [36].

Some Java APIs, like core API and XML API, are also integrated into the platform. These APIs are essential for DTV application developments. The lifecyle of each application is maintained by a resident Application Manager, which utilizes Xlet lifecycle model. In

addition, Navigator and Digital Teletext are also implemented as resident applications. Figure 16 shows the architectural overview of the implementation.

5.3 Architecture Design and Implementation

The goal of this thesis is to design a high level Java 3D rendering API to STB platform. Due to the limited computing resources and the uniqueness of DTV environment, some compromises have to be made. For example, the majority of TV viewers are not as sensitive to image quality as PC game players, and TV is usually viewed at such a distance that some details are not easily recognizable. However, TV viewers are very sensitive to the speed of rendering, and pictures rendered in low frame rates will undermine the viewing experience dramatically. As a result, rendering speed is upmost important in DTV environement, and sometimes rendering quality could be sacrificed to achieve real-timeness. Another issue need to be addressed is the balance between minimizing API size and maximizing feature sets. To deploy in a digital STB, the 3D API needs to be compact. On the other hand, the API should support a complete set of 3D features, which requires a lot of memory spaces. This kind of contradiction requires careful design to pack as many features as possible into limited resources.

5.3.1 Java and 3D Rendering API

In HUT’s STB reference implementation, Java is selected as the application programming language. The advantages that Java technology brings to embedded system are discussed in Chapter 3, and some DTV standards, like MHP, are especially rolled out based on Java. As Java3D is the standard high-level 3D API in Java platform, the most likely question would be: why do we need another Java 3D API in STB?

There are four reasons to justify the necessity of another dedicated Java 3D API in STB environment. First, Java3D is basically not a rendering API – it uses OpenGL or Direct3D to do the low-level rendering. Rather, it defines a set of high level API to manage the scene graph and basic 3D vector maths. The extra API makes Java3D library quite big, and thus unsuitable for the limit memory space of a STB. Second, though OpenGL core is



independent of any window system, the rendering context needs to hook to native window system (like X-window in Linux) to facilitate interaction and event handling. For example, GLX library in Linux is used to connect OpenGL core and X-window. Java3D also has to rely on the native window system to interact with OpenGL rendering context. However, in HUT’s STB implementation, X-window subsystem is removed to satisfy the memory requirement, and all the Java controls are rendered to the framebuffer directly. As a result, it is not feasible to use Java3D in this particular environment without X-window support. Third, Java3D implements lots of 3D vector calculation in Java level to ease the transform between different coordinate systems, which will inevitably affect the performance of the whole 3D pipeline. Last, though it uses OpenGL or Direct3D as rendering API, Java3D is a totally different API requiring deverlopers to learn and get familiar with, which increases the cost when migrating OpenGL or Direct3D applications to Java3D.

All these issues indicate Java3D is not suitable for STB environement, and a compact, fast, easy to learn 3D API is needed. In particular, this API should be able to work with framebuffer directly without any extra window system support.

To make a framework meeting these requirements, selecting a compact and complete low-level rendering API is crucial. OpenGL ES API [37] is a subset of the standard OpenGL API for embedded device with limited hardware resources, and it keeps the most efficient functions in OpenGL library, like glPointer series functions. In addition, it is complete, which means it can do virtually any OpenGL operations with its own set of functions. It is consequently the ideal candidate for low-level rendering API in STB environment.

5.3.2 Java Native Interface

JNI is a framework for Java applications to access to native operations and resources in operating system level. JNI plays an important role in extending Java functionality, when standard Java class library does not support platform-dependent features. As glue between Java and native methods, JNI provides an elegant solution to port native libraries or applications into Java, and these libraries or applications are written in other languages like C and C++.

Figure 17. JNI is the bridge between TVGL and OpenGL ES.



TVGL also uses JNI to bind the Java API layer with native OpenGL ES layer. Figure 17 shows the relationship between TVGL, JNI and OpenGL ES. Applications call TVGL API functions, and TVGL forwards the call to JNI framwork immediately. JNI finally maps the TVGL function to the native OpenGL operation; in addition, JNI also transfers application data (vertex position, normal, color and so on) from TVGL to OpenGL ES and vise versa.

5.3.3 Interface Defintion and Intergation with DTV Platform

TVGL needs to be compact, complete and easy to learn. To this end, the interface of this framework should be simple and close to that of low-level OpenGL ES API. In TVGL, there is a separate class, called GL, dedicated for wrapping all the native OpenGL operations, and the name of these operations in TVGL is identical to the ones in OpenGL standard. As a result, applications based on native OpenGL ES API could be migrated into Java platform with little efforts – developers do not even need to change the signature of OpenGL operations.

All OpenGL operations are managed by GL context, which is basically a state machine accumulating 3D operations procedurally. Procedure-oriented programming might be quite unfamiliar to Java programmers, because Java is inherently an object-oriented programming language. To make procedural OpenGL operations accessible in an object-oriented way, TVGL defines adaptor classes: GLRenderer and GLObject. GLRenderer is the abstract class for rendering opertions, and it generalizes all 3D operations into one method – render(). Another method, control(), abstracts the interactions between rendering context and user interface. All 3D renderers based on TVGL should be deriving from GLRenderer and implement their own control and render method. GLObject is the collective class for a renderable object, which defines the data needed to describe a 3D object, and deriving from this class will generate a variety of different renderable entities. The diagram of these classes is shown in figure 18.

TVGL based 3D applications should be very easy to integrate with existing DTV applications in HUT’s reference implementation. A common use senario is that 3D graphics is rendered along with other DTV 2D widgets, and users control the 3D content through 2D widgets, like

OpenGL ES operations are mapped in GL.

GLGLObject

renderer()

EventLis tener

GLRenderer

<<abstract>> render()<<abstract>> control()

HaviWidget

from DTV package

Figure 18. Diagram of TVGL classes.



buttons and sliders. In HUT’s DTV package, Havi widgets are capable of managing a screen window. By setting the rendering target to a particular Havi widget, GLRenderer can render to a window and integrate with other DTV applications seamlessly.

Another integration issue is event handling, by which a user can control and manipulate the 3D content. GLRenderer utilizes DTV package’s event listener mechanism to listen to user’s input. By implementing the control method, GLRenderer can manipulate the content in a renderer-specific way.

5.3.4 Implementation Issues

During the implementation, some issues have to be carefully considered to make a usable 3D API library under limited hardware resources. The highest priority is efficiency, and special measures have been taken to improve the performance of the framework. On the other hand, the performance might be compromised due to some technical limitations. This section will discuss these issues in detail.

• OpenGL ES and Mesa3D: OpenGL ES is a new standard proposed by Khronos group, and the first specification – OpenGL ES 1.0 -- was released in July, 2003. [23] However, there is no publically available reference implementation for OpenGL ES API, though some mobile 3D graphics vendors claimed their priopretary libraries were OpenGL ES compatible.

It is unnecessary and unpractical to implement the native OpenGL ES API in this thesis, because mobile 3D graphics chip manufactures will eventually release their implementation based on hardware acceleration. Given the fact that OpenGL ES is a subset of OpenGL 1.3, we can select one OpenGL1.3 implementation as a replacement of OpenGL ES. In future, if OpenGL ES implementation is available, this implementation could be easily replaced.

TVGL selected Mesa3D [24] as the native rendering API, as a replacement to OpenGL ES. Mesa3D is an open sourced 3D library supporting OpenGL 1.3, and it also provides a pure software implementation for OpenGL operations, which means it works on 3D graphics hardware from different vendors. Certainly, doing 3D operations in software mode compromises the performance of 3D pipeline, and Mesa3D also makes it possible to configure some 3D operations running in hardware mode depending on the graphics card capabilities.

• Offscreen rendering and SDL surfaces swaping: HUT’s STB reference implementation requires Mesa3D to render to framebuffer directly. However, by default, Mesa3D only supports rendering to a window managed by native window subsystem, and the rendering context is hooked with a particular window to retrieve the user input events. To make it work with STB environement, GL rendering context needs to be hooked with a subsection of the framebuffer. OSMesa is an utitility library included in Mesa3D, and it has the capability to configure Mesa3D to render the content to a system memory buffer in offscreen mode.

SDL surface, basically a memory area, is the canvas where HAVi widgets are rendered, and 3D content should be also rendered into a SDL surface in order to work



with HAVi widgets. The content of OSMesa rendering buffer then needs to be copied into SDL surface after the 3D rendering is done for each frame, which might be the bottleneck of realtime rendering because massive amount of memory needs to be copied in 1/20 – 1/30 seconds. To make things worse, offscreen rendering in Mesa3D cannot utilize any hardware acceleration capability of the graphics card, and all 3D calculations have to be done in CPU. All these limitations indicate that TVGL probably might be only suitable for simple 3D scene rendering with less than ten thousand polygons and vertices.

To achieve the maximum performance, an application needs to call as minimum 3D operations from Mesa3D as possible. OpenGL ES API drops out all the ‘slow’ operations in OpenGL specification. For example, the oprations controlling one single vertex like glVertex3f are not included in OpenGL ES any more. TVGL also adopts this philosophy, and the user of TVGL library is forced to use ‘fast’ 3D operations like vertex array and index array.

• Fixed point and floating pointer implementation: OpenGL ES specification also defines a set of fixed-point 3D operations, to make 3D graphics accessible for low-end devices that do not have floating-point capabilities. TVGL does not implement the fixed-point functions as defined in OpenGL ES, due to the fact that most STB platform has intrinsic floating-point support.

Fixed-point implementation certainly runs faster than floating-point implementation, and less data needs to be copied from Java layer to native OpenGL ES layer. Fortunately, it is straightforward to extend TVGL to support fixed-point implementation in the future, when fix-point OpenGL ES implementation is available.

5.4 Future Improvements

TVGL fulfills all the basic requirements of a 3D API in STB environment; it is compact – only about 100 3D functions are included; it is complete – all the standard OpenGL operations can be implemented with TVGL functions; it is easy to learn – programmers with OpenGL experience can get familiar with TVGL immediately, because the functions in TVGL are indentical to OpenGL functions. With TVGL, virtually any 3D applications based on OpenGL could be migrated to digital STB environment with minimal efforts.

However, a widely accepted API requires intensive redesign and testing, which is unlikely for TVGL in the period of this thesis. Some improvements could be made in the future to extend the functionalities of TVGL and improve the performance of 3D rendering. The following sections discuss these improvements.

• Native OpenGL ES library: The current TVGL implementation uses Mesa3D as a temporary solution to native OpenGL ES library. Because Mesa3D is initially designed for PC, it does not take into account the situations under STB environement. In addition, though Mesa3D works with most PC graphics cards and is able to utilize hardware acceleration capability of most commercial PC graphics chips, it is unlikely to support a variety of STB graphics hardwares. Under this circumstance, 3D calculation has to be done in software mode.



TVGL could archive great performance improvement if the native OpenGL ES library can utilize the hardware acceleration of graphics card. STB manufacturers start to integrate 3D acceleration capability to STBs, and they will either release their own OpenGL ES library customized for their products or provide drivers that are compatible to reference OpenGL ES implementation. When these implementations are available, we can easily replace the native Mesa3D library with a specific OpenGL ES library, while keep the high-level TVGL API intact.

• Advanced offscreen rendering: Offscreen rendering is essential to make TVGL applications work with Havi widgets, without any native window subsystem support. Currently, the 3D content is rendered to a system memory buffer, which is then copied into the SDL surface. But, this means Mesa3D cannot use any hardware accelerated 3D operation at all. Moreover, copying from system memory to video memory (SDL surface) is a slow operation.

There are some alternatives to OSMesa offscreen rendering, and they could improve the performance by utilizing 3D hardware capacity or reducing memory to be copied between system and video memory. The first method is rendering to texture, where a 2D texture (usually a video memory area) is used as the rendering target. The rendered texture can then be blitted to a screen area directly. With this method, 3D hardware acceleration could be used and it would be much faster than OSMesa solution. However, rendering to texture itself needs hardware support, and if the hardware does not have this feature, system memory will be used as the rendering target and performace will be compromised.

Another method is to write a native utility library to connect OpenGL ES with SDL surface, i.e., the role of this library is exactly the same as GLX in Linux and WGL in Windows. This will solve all the problems like memory copying and lack of 3D hardware support with offscreen rendering. Though implementing such a library requires several man-months’ work, it is worthwhile if we want to implement a stable and efficient 3D framework.

• A full-fledged OO 3D graphics engine: TVGL defines two fundamental classes to help programmer using the library in an object-oriented way. Nevertheless, these two classes just abstract the very basic concepts in a 3D application: object and renderer. A 3D application might need complex scene management, key-frame management and user event management, and these features are common to all 3D applications. As a result, it is desirable to make TVGL a complete object-oriented 3D graphics engine, which will eventually facilitate the development of 3D applications.

In addition to the features of a 3D graphics engine, TVGL in future should also include tools and utilities that are convenient for 3D application development. For instance, 3D contents are usually first edited in commercial 3D software, like 3Dmax, Lightwave, and TVGL should provide data loaders to import the 3D model from these applications and display it in TV screen.



5.5 Case Study

Several applications are developed to demonstrate the functionality of TVGL API, and these applications could also be used as tutorials for developers to get familiar with the framework. In addition, developers could use these applications as skeletons when designing their own TVGL-based applications.

Though TVGL is designed in mind to work with HUT’s STB reference implementation, it is meaningless to develop the demo applications on this particular environment. The most obvious reason is this implementation was proposed over 2 years ago, and the hardware is outdated comparing to current STB configurations. For example, according to the feasibility study in chapter 4, some mainstream set-boxes have a 500Mhz CPU, which is twice the frequency of HUT implementation. It is unlikely to make a visually appealing real time 3D demo application on this platform, and a more capable system is needed.

Fortunately, Ubik can be used in any Linux-based PC, the hardware of which could be easily configured and customized, and a real STB is not needed to do the development and testing. For these demos, a Linux PC with Intel Pentium III 933Mhz CPU and 256MB memory is used as the test bed, and the result is roughly scaled down to estimate the performance in a system with 500Mhz CPU. Rough downscaling makes sense because all 3D computation in TVGL is done in the CPU, and we could assume the performance of a CPU is in proportial to its frequency. In reality, a STB might be faster than a PC with the same CPU frequency and memory capacity, because CPU and memory bus of STBs are usually optimized.

5.5.1 Demo Environment

All the demos are running in a Linux-based PC with the hardware configuration as follow:

• CPU: Intel Pentium III 933MHz

• Memory: 256MB

• Video Card: Matrox Millennium G450 with 16MB video memory

And the software configuration is the same as the reference implementation, which includes:

• Operating System: Linux without X-window system, SDL with framebuffer support

• Java Virtual Machine: Kaffe, using console framebuffer for rendering widgets

• Native OpenGL Library: Mesa3D, with OSMesa extension for offscreen rendering

• TVGL: TVGL library

• Java APIs: core Java API, JMF

• DTV Applications: Navigator, Digital Teletext



5.5.2 Demo Applications

As mentioned previously, virtually all OpenGL-based applications can be easily migrated into TVGL environment. The purpose of these demos is to show the functionalities of TVGL and to prove that the whole API works. Consequently, these demos are focused to show how to use the library and organize applications based on TVGL, not to teach developers how to implement 3D graphics applications with nice visual effects, because these techniques could be learned from abundant OpenGL tutorials and books. There are two demo applications: cube and gears.

• Cube: Cube is used to demonstrate how easy it is to write a simple 3D application based on TVGL. The colored cube consists of 12 triangles and 8 vertices, and it is rotating in a hardcode way, i.e., user cannot control the movement of the cube. Figure

19 is a screenshort of this demo.

• Gears: Gears is a comprehensive demo that includes most of the features of TVGL. Gears is migrated from Mesa3D gears demo directly, with little modification. Therefore, Gears also shows how to migrate an OpenGL based application to TVGL. The geometry needs to be reconstructed, because Mesa3D gears demo uses quad strips, but TVGL only support triangle lists. All the geometry operations on single vertex should also be replaced by vertex arrays and index arrays in TVGL, to minimize the amount of calling to native Mesa3D library. Gears also implements simple event handling, with which users can interact with the 3D model.

Gears could also be used as a benchmark program for 3D performance in STB environment, because the complexity of geometry could be controlled in the code. For example, the number of triangles and vertices could be easily increased by tuning the parameters of geometry functions, and the throughput of the 3D pipeline could be measured. In the demo application, Gears has 1360 vertices and 2160 triangles. A screenshort of this demo is shown in figure 20.

Figure 19. A screen shot of 3D cube and AWT widget.



5.5.3 Results Analysis

The result of these demos needs to be carefully analyzed to check if TVGL fulfills the requirements to work on digital STB environment. The first analysis to be done is the memory overhead caused by TVGL. The native binding functions of TVGL are implemented along with other native functions of SDL library, the size of which is increased by 12KB. However, the memory size of native OpenGL ES library could not be determined and we could not use the size of Mesa3D library, because TVGL just used a subset of Mesa3D. It could be predicted that the future OpenGL ES implementation takes less than 500KB, because it needs to be working on embedded systems like mobile phones. The following table summarizes the memory size of each component in TVGL.

Table 3. Memory usage of TVGL components.

Componet Memory size OpenGL ES library <500KB (estimated) Native JNI binding functions 12KB tvgl.jar 14KB

Another interesting analysis is the performance of TVGL library, and we need to know if it is really possible to do real-time 3D rendering in STB environment. Usually, the complexity of a 3D scene is measured by the amount of polygons and vertices it has, and the most straightforward criteria for 3D rendering performance is the framerate the pipeline can reach when rendering a scene with certain complexity. Gears demo provides a simple benchmarking program for performance evaluation. The following table shows the framerates when 3D scenes of different complexities are rendered with TVGL, and the estimated downscaled result is also listed in the last column.

Figure 20. A screenshot of Gears with AWT widget.



Table 4. Framerates when rendering scenes with different complexity.

Triangles Vertices Framerate in test machine Downscaled framerate in 500Mhz CPU

1360 2160 82FPS 44FPS 2720 4320 72FPS 39FPS 5440 8640 57FPS 31FPS 8160 12960 47FPS 25FPS

The resolution of rendering target also has effect on performance, because it decides the number of rendered pixels and the memory amount to be copied. A small test is also done to investigate the relationship between framerate and rendering resolution. By changing the target resolution while keeping the complexity (1360 triangles and 2160 vertices), we got the results as the following talbe shows.

Table 5. Framerates when rendering scenes with different resolution.

Resolution Framerate in test machine Downscale framerate in 500Mhz CPU

256x256 82FPS 44FPS 320x320 60FPS 32FPS 480x480 30FPS 16FPS

The static and performance analysis reveals that TVGL is suitable for rendering simple 3D scene with less than 5000 triangles. This level of complexity should be sufficient for most 3D meshes, when there were viewed from 2-3 meters with a resolution of 300 by 300. However, intensive usability testing needs to be done to understand how sensitive TV viewer’s eyes are when watching 3D content on TV screen, and TVGL could be improved to archive maximum visual satisfication with minimum computational resources.


Conclusion 44

6. CONCLUSION

DTV will undoubtly replace traditional analog television, because DTV achives higher frequency efficiency than analog TV, offers broadcasters the possibility to provide sharper and clearer images, and above all, presents TV viewers with a wide variety of interactive services that are impossible in analog television. As an important form of entertainment, 3D graphics applications will make its way in DTV environment.

Digital STB is the device that executes all DTV applications and services, and it has a similar architecture to PC. Java technology plays a crucial role in broadcasted DTV applications development, due to the fact that Java is platform independent. Java technology provides a comprehensive set of APIs that could be used to develop DTV applications and services independent of targeting platform.

3D graphics applications are computational demanding, requiring high-speed CPU, large amount of memory and even a 3D acceleration card. Consequently, it is very challenging to introduce 3D graphics in STB platform because its computing resources are very limited. Fortunately, a feasibility study reveals the latest STBs are capable of rendering lightweight 3D graphics with the complexity close to PC games in 1998-1999, and the purpose of this thesis is to develop a basic API that could be used to implement Java-based 3D applications in digital STB environment.

TVGL, the 3D Java API in this thesis, is compact, complete and easy to learn. It has the simplest architecture and also defines a framework, based on which complex 3D applications could be constructed. TVGL applications could also be integrated with other DTV applications seamlessly. Two demos are implemented to show the functionality and usage of TVGL, and these demos also prove TVGL fulfills all the basic requirements of a 3D API working in digital STB environment. In addition, TVGL could be improved in furture to achieve better performances, when hardware acclerated native rendering library is available, and evolving TVGL to a comprehensive graphics engine will facilitate the development of 3D applications dramatically.

Up to now, 3D graphics is not specified as a requirement at current DTV standards, because STBs in mass market are computationally limited and there is no solid business case for 3D applications. However, the computing capability of digital STBs keeps improving and some pioneer manufacturers start to integrated 3D graphics chip in their digital set-box products. It is believed 3D graphics applications will be widely accepted by TV viewers. TVGL, as a pilot 3D API in STB environment, might draw the attention of DTV standard makers and convince them that 3D graphics in digital STBs is feasible and will bring great opportunities to the DTV industry.

The ultimate goal of TVGL is to evolve as a 3D API standard in digital STB environements. For example, TVGL could extend and enrich MHP standard by adding a new profile for high-end STBs capable of doing realtime 3D graphics. Applications developed with this standard API could be distributed and run in any STB of the new profile. To achieve this goal, there is still a long way to go and great improvements needs to be made.


References 45

REFERENCES

[1] Booth S.A., “Digital TV in the USA,” IEEE Spectrum, Volume: 88, Number: 3, pages: 39-46, March 1999.

[2] Canal Digital, “Technology,” referred on 17.06.2003, available at www.canaldigital.com/tech.asp.

[3] Sandbank C. P., “Digital Television in the Convergent Environment,” IEEE Computer Graphics and Applications, Volume: 21, Issue: 1, pages: 32-36, January-February 2001.

[4] Robin M., Poulin M., “Digital Television Fundamentals: Design and Installation of Video and Audio Systems,” McGraw-Hill, 1997.

[5] Henriksson J., “DVB-X,” referred on 18.06.2003, available at http://www.dvb.org/documents/white-papers/DVB-X article from DVB Scene 05.pdf

[6] Murakami T., et. al, ”Overview: The Present and Future of Digital TV Broadcasting,” referred on 18.06.2003, available at http://global.mitsubishielectric.com/pdf/advance/vol85/85tr2.pdf

[7] Nordig, “Nordig Unified Requirements: Nordig Unified IRD specification, ver1.1,” referred on 20.06.2003, available at http://www.nordig.org/pdf/NorDigUnified-Compliance100.pdf

[8] Odenwald D., “Stations to Digital TV Data: Drop Dead!,” referred on 19.06.2003, available at http://www.current.org/dtv/dtv0120.html

[9] Fox B., “Digital TV Rollout (US Digital Terrestial TV),” IEEE Spectrum, Volume: 35, Issue: 2, pages: 65-67, February 2001.

[10] O’Driscoll G., “The Essential Guide to Digital Set-top Boxes and Interactive TV,” Printice Hall PTR, Ireland, 2000.

[11] Datta T., “Digital Covergence: the Set Top Box and Its Digital Interface,” White paper, Enthink, July 1998.

[12] Hurley T.R., “Evolution of the Digital Set-top Box,” International Broadcasting Convention, (CONF. Publ N. 428) pages: 277-282, 1996.

[13] Nilson J., “Usability Engineering,” Morgan Kaufmann, San Francisco, 1994.

[14] Gist Communications, “Gist Communications – Interactive TV Guide,” referred on 22.06.2003, available at http://www.gist.com/us/products/programguides.html

[15] Peng C., “Digital Television Applications,” Doctoral Dissertation, Helsinki University of Technology, Espoo, Finland, November 2002.

[16] ABC TV, “ABC TV – Long Way to the Top,” referred on 06.07.2003, available at http://www.abc.net.au/longway/concert/itv.htm


References 46

[17] Bird J., “Digital Television Broadcasting: Perspectives on the Furture,” Master’s Thesis, Swinburne University of Technology, Australia, 2002.

[18] Net4TV, “Net4TV Voice: First Look at AOLTV,” referred on 07.07.2003, available at http://net4tv.com/voice/story.cfm?storyid=3056

[19] DataMonitor, “iTV Games and Gambling in Europe: Service and Business Model Development to 2007,” Market Report, February 2003.

[20] G-Cluster, “System Architecture,” refered on 07.07.2003, available at http://www.g-cluster.com/technology/index.html

[21] Mathews M., Marc R. E., “Embedded Linux & Java – Made for Each Other,” refered on 14.07.2003, available at http://www.linuxdevices.com/articles/AT6624052896.html

[22] Venners B., “Inside the Java Virtual Machine,” McGraw-Hill, 2000.

[23] Calder B., Courney J., etc., “Java TV API Technical Overview: The Java TV API Whitepaper version 1.0,” White paper, Sun Microsystems, 2000.

[24] Peising J., Löytänä K., ”The DAVIC API for Interactive TV,” Technical presentation, referred on 18.07.2003, available at http://www.davic.org/Info_Day/piesing.pdf

[25] DVB, “DVB-MHP – What is MHP?,” referred on 20.07.2003, available at http://www.mhp.org/what_is_mhp/index.html

[26] López J.C., López C., etc., “A MHP Receiver Over RT-Linux for Digital TV,” IADIS International Conference, WWW/Internet 2002, Lisbon, Portugal, November 13-15, 2002.

[27] Perrot P., “DVB-HTML – An Optional Declarative Language within MHP,” White paper, referred on 12.08.2003, available at http://www.mhp.org/what_is_mhp/pdf_and _other_files/trev_288-perrot.pdf

[28] MHP, “DVB-MHP- What is MHP?,” referred at 18.07.2003, available at http://www.mhp.org/what_is_mhp/index.html

[29] Smith-Chaigneau A., “DVB-MHP – A Snapshot,” Technical presentation, referred on 20.07.2003, available at http://www.broadcastpapers.com/data/DVBMHPSnapshot - print.htm

[30] Nvdia Corporation, “Geforce FX 5900,” referred on 06.09.2003, available at http://www.nvidia.com/page/fx_5900.html

[31] OpenGL, “Overview of OpenGL,” referred on 12.08.2003, available at http://www.opengl.org/developers/about/overview.html

[32] Lime Studios, “Lime Studios/solutions/limePlay-how,” referred on 06.09.2003, available at http://www.limestudios.com/solutions/limePlay-how


References 47

[33] Nokia, “Nokia Mediamaster 9902 S,” referred on 04.08.2003, available at http://www.nokia.com/nokia/0,8764,3591,00.html

[34] ATI Technology Inc., “XILLEON 220,” referred on 04.08.2003, available at http://www.ati.com/products/xilleon220/index.html

[35] VIA Technologies, Inc., “VIA Set Top Box Reference Design,” referred on 07.08.2003, available at http://www.via.com.tw/en/VInternet/settopbox.jsp

[36] Peng C., Cesar P., Vourimaa P., “Integration of Applications Into Digital Television Environment,” International Workshop on Multimedia Technology, Architecture, and Application, Taipei, Taiwan, pages: 266-272, 26-28 September, 2001.

[37] Peng C. and Vuorimaa P., “Interactive Digital Teletext Service,” Proceedings of the 6th World Multi-conference of Systemics, Cybernetics and Informatics, Orlando, Florida, pages: 181-186, 2002.

[38] Khronos Group, “OpenGL ES Common/Common-Lite Profile Specification,” referred on 15.08.2003, available at http://www.khronos.org/cgi-bin/fetch/fetch.cgi?opengles_spec

[39] Gordon R., “Essential JNI: Java Native Interface: Programmer’s Guide and Specification,” Prentice Hall PTR, 1998.

[40] Vuorimaa P., ”Digital Television Service Architecture,” Proceeding of the IEEE International Conference on Multimedia and Expo, ICME2000, New York, 2000.

[41] Fox B., “Digital Television Comes Down to Earth,” IEEE Spectrum, Volume: 35, Issue: 10, pages: 23-29, October 1998.

[42] Milenkovic M., “Delivering Interactive Services via a Digital TV Infrastructure,” IEEE Multimedia, Volume: 5, Issue: 4, October-November 1998.

[43] Evain J.P., “The Multimedia Home Platform – An Overview,” EBU Technical Review, 1998.

[44] Wood D., “The DVB Project: Philosophy and Core System,” Electronics and Communication Journal, Volume: 9, Issue: 1, pages: 5-10, February 1997.

[45] Reimers U., “Digital Television Broadcasting,” IEEE Communication Magazine, Volume: 36, Issue: 1, pages: 104-110, Feburary 1998.

[46] Ciciora W. S., “Inside the Set-top Box,” IEEE Spectrum, Volume: 33, Issue: 4, pages. 70-75, April 1995.

[47] Pekowsky S. and Jaeger R., “The Set-top Box as Multi-media Terminal,” Transactions on Consumer Electronics, Volume: 44, Issue: 3, , pages: 833-840, August 1998.

[48] Droitcourt J.-L., “Integrate Architecture-anatomy of the Interactive Televsion Set-top Box, How it Works, and what it means to the consumer,” International Boradcasting Convention, pages: 272-276, 12-16 September 1996.


References 48

[49] Droitcourt J.-L., “Understanding How Interactive Television Set-top Box Works, and what it will mean to the customer,” International Broadcasting Convention, pages: 382-394, 14-18 September 1995.

[50] Nave R. and Rsuria Y., “Advanced Use of TV Set top Boxes,” International Broadcasting Convention, pages: 436-440, 12-16 September 1997.

[51] Peng C. and Vuorimaa P., “Developement of JAVA User Interface for Digital Television,” Proceedings of the 8th International Conference in Central Europe on Computer Graphics, Visualization, and Interactive Digital Media, WSCG'2000, Plzen, Czech Republic, pages: 120-125, 2000.

[52] Peng C. and Vuorimaa P., “A Digital Television Navigator,” Proceedings of the IASTED International Conference on Internet and Multimedia Systems and Application, Las Vegas, Nevada, pages: 69-74, 2000.

[53] Peng C. and Vuorimaa P., “A Digital Teletext Browser,” Proceedings of the 9th International Conference in Central Europe on Computer Grahics, Visualization, and Interactive Digital Media, WSCG'2001, Plzen, Czech Republic, pages: 120-125, 2001.

[54] Peng C. and Vuorimaa P., “Digital Television Application Manager,” Proceedings of the IEEE International Conference on Multimedia dn Expo, ICME2001, Tokyo, Japan, pages: 685-688, 2001.

[55] Cesar P., “Havi Components in Digital Television,” Master’s thesis, Helsinki University of Technology, November 2001.

[56] Sivaraman G., Cesar P., Vourimaa P., “System Software for Digital Television Applications,” IEEE International Conference on Multimedia and Expo, Tokyo, Japan, Pages: 784-786, August 22-25, 2001.

[57] Tannenbaum D., “A Java Technology Overview for the Embedded Linux Market,” referred on 18.08.2003, available at http://www.linuxdevices.com/articles/AT7996122293.html

[58] Watt A. H., “3D Computer Graphics,” Pearson Addison Wesley, 3rd edition, December 1999.

[59] P. Cesar, J. Vierinen and P. Vuorimaa, “Open graphical framework for interactive TV,” accepted to the 5th International Symposium on Multimedia Software Engineering, MSE2003, Taichung, Taiwan, December 10-12, 2003.


Implemented OpenGL ES Functions 49

APPENDIX A: IMPLEMENTED OPENGL ES FUNCTIONS

The following table lists OpenGL ES functions that are implemented in TVGL. In this table, the implemented functions are listed in first column. To help developers porting existing OpenGL applications to TVGL, the second column lists the OpenGL functions that could be replaced by the function in first column of the same row.

Function Replaced OpenGL functions glActiveTexture glAlphaFunc glBindTexture glBlendFunc glClear glClearColor glClearDepthf glClearStencil glClientActiveTexture glColor4f glColor3(bdfis) glColor4(bdis) functions glColorMask glColorPointer glColor3(bdfis)v glColor4(bdis)v functions glCompressedTexImage2D glCompressedTexSubImage2D glCopyTexImage2D glCopyTexSubImage2D glCullFace glDeleteTextures glDepthFunc glDepthMask glDepthRangef glDisable glDisableClientState glDrawArrays glDrawElements glEnable glEnableClientState glFinish glFlush glFogf glFogi glFogfv glFogiv glFrontFace glFrustumf glGenTextures glGetError glGetIntegerv glGetString


Implemented OpenGL ES Functions 50

glHint glLightModelf glLightModelfv glLightf glLighti glLightfv glLIghtiv glLineWidth glLoadIdentity glLoadMatrixf glLoadMatrixd glLogicOp glMaterialf glMateriali glMaterialfv glMaterialiv glMatrixMode glMultMatrixf glMultMatrixd glMultiTexCoord4f glNormal3f glNormal3(bdis) functions glNormalPointer glNormal3(bdis)v functions glOrthof glOrtho glPixelStorei glPointSize glPolygonOffset glPopMatrix glPushMatrix glReadPixels glRotatef glSampleCoverage glScalef glScalex glScissor glShadeModel glStencilFunc glStencilMask glStencilOp glTexCoordPointer glTexEnvf glTexEnvfv glTexImage2D glTexParameterf glTexSubImage2D glTranslatef glVertexPointer glVertex2(dfis)(v), glVertex3(dfis)(v), glvertex4(dfis)(v) glViewport

Java_3D_stb

Documents

Transcript of Java_3D_stb