The MegaStore Application

A. Benabdelkader ©UvA, 2002

The MegaStore Application

Advanced Internet-based Electronic Commerce Technologies for Music Industry

Databases and Web-Based Applications

Ammar Benabdelkader

University of Amsterdam

December 1, 2003

UvA, Amsterdam


• Large number of emerging e-commerce applications. These applications comprise large multimedia data, for which:

– Data classification and cataloguing are required

– Related information needs to be properly inter-linked

– Efficient storage, and access of large multimedia data is supported

– Short response time for on-line requests must be guaranteed

– High data transfer rates must be provided

• Furthermore, public information must be separated from the private information that need to be securely kept at the place where it belongs.

Motivation


• An efficient architecture must be designed addressing the need to provide the user of electronic commerce with an environment through which he can experience as sufficiently close to real life shopping environment. The approach considers the following aspects:

– Security for data access and users authentication

– Suitable user friendly interfaces

– Performance issues must be addressed

– Private data need to be protected

– Public data is made available to the Internet users

• The MegaStore system aims at the design and set-up of the necessary database structure and platform architecture for advanced e-commerce applications

Approach


• From the analysis of music industry application, the following aspects are identified:

– The data is geographically distributed over the network

– Information about music is classified into two main categories:

• The general information stored at the directory services (database catalogue) and accessible by all the Internet users

• The raw music data that can only be accessible by the music storekeepers at music centers or burning towers

– Depending on the user profile and authorization, only a part of the information can be accessed and users need not to know about the data distribution

– The real music data must be securely transferred through a dedicated network

– High bandwidth connection is necessary to handle raw music data that need to be transferred between the music storage centers and the burning towers

– Low latency network connection is necessary to support the huge number of users expected to connect to the system

Problem Analysis


Database Design

SONG

Song ID: ShortSong Title: StringSong Type: StringSong Genre: StringRelease Date: DateSong Lyrics: StringSong Price: RealStatus Code: CharSong Label: StringSong Barcode: ShortSort Code: CharSong Prefix: StringSong Suffix: StringSong Flags: StringSong Duration: TimeSong Language: StringSong Stream: AudioSong Video: Video

Album

Album ID: ShortAlbum Title: StringAlbum Publisher: StringAlbum Cover: ImageAlbum Price: Float Album PubYear: Date

Order

Order ID: ShortOrder Date: Date

Customer

Cust ID: ShortCust Name: StringCust Faithful: ShortCust Company: StringCust Address: StringCust Country: StringCust Email: String

Instrument

InstName: StringInstCat: String

1..*

ArtistAlbums AlbumArtists

0..*AlbumOrders

OrderAlbums

SongOfAlbum

AlbumSongs

0..*

OrderedBy CustOrders

Used InUses

SungBy

ArtistSongs

0..*

0..*

1..*

0..*

1..*1..*

1..*

1..1

0..*

ARTIST

Artist Name: StringArtist Bio: StringArtist Language: StringArtist Image: Image

Composer Performer


The MegaStore System Architecture (1)

The designed system architecture involves the following components:

– The back-end system, including the database engine and the predefined networking connection between the MegaStore system components

– The front-end system, including:• The Internet–Shop interface, where a user from home (or

work place) can search for music, listen/watch to the audio/video clips, and order CDs, and

• The Shop-in-a-Shop interface, where the music storekeeper can fetch on-line the real music data from its original source in order to burn at run-time the requested music CDs


The MegaStore System Architecture (2)

Parallel Distributed Database Server

Local database

Web Server

Local database

Web Server

Local database

Web Server

Local database

Web Server

I n t r a n e t

LAN

Directory Services

Database Catalogue

Internet Shop

Interface

Internet Shop

Interface

High bandwidth

Low latency

medium bandwidth- medium latency

Internet

Remote Applications

WWW Browser

Remote Client

WWW Browser

Shop-in-a-shopInterface

Shop-in-a-shopInterface

Shop-in-a-shopServer

Internet


Activity Diagram for the Internet-Shop Interface

UserLogin

ShowHelp

ArtistEntry

SongEntry

AlbumEntry

ShowAlbums

AlbumSongs

Orderalbums

CustomOrders

ShowSongs

SongArtists

ShowArtists

ArtistSongs

S

InternetUser

Level 1

Level 2

Level 3


Internet Web Interface (1)


Internet Web Interface (2)

(a) Song Search (b) Album Search


Dynamic Browsing of Interface

(A) Artist Songs

(B) Album Songs

(B)(A)


MegaStore Ordering System

Submitted

OrderCreated

Handled

OrderHandling

Logged

UserLogging

OrderArchived

(a)Standard Order

(b)Custom Order

State Diagram for Orders


System Implementation

• The database schema is implemented on top of the Matisse object-oriented

database system

• The Matisse back-end database runs on a cluster composed of 20 nodes.

• The MegaStore Internet interface is implemented using an NT front-end

machine, that is in turn connected to the underlying back-end database.

• The Internet-shop server prototype is implemented using a combination of

the following software technologies:

– JavaScript and Vbscript for tips programming,

– Active Database Objects (ADO) for database access, and

– HTML for text formatting

• the server implementation is made possible using the Active Server Pages

(ASP) environment that allows the combination of different software

technologies in one single environment.


Databases

- ODL is used to support the portability of database schemas across conforming ODBMSs.

- OIF is used to exchange objects between databases and provide database documentation.

Applications

- Universal data access through standards and middle ware solutions (XML, ODBC, JDBC, JAVA, etc.).

F- B

Legacy systems (Flat-files and Databases)

Database access

- ODL- OIF

ODBMS

Adapter

C++

JAVA

SQL

SQL3

Adapter Adapter

Object-orientedUser Interfaces

DBA Program

Access independent Framework

Facilitating Applications

- ODBC- XML- JDBC, Java

Information Management Strategy

F- CF- A


Management of Large Multi-Media Data

• An efficient architecture must be designed for manipulation of large data sets. Therefore, Data management mechanisms must be addressed in such a way that:– Data needs to be properly searched, retrieved, published, inter-

linked, and compared to other data,

– Information security is preserved,

– Performance issues are improved,

– Private data is protected, while published data is made available to the outside users.


• File System Approach and HFS (Hierarchical File System)

• External Data Link Approach

– database catalogue refers to NFS (network file system)

• One-Database Storage Approach

• Parallel/Distributed Database Server

• etc.

Data Storage Approaches


• Inefficient in:

– Maintaining the link between the inter-related pieces of information,

– Comparing related data in different applications,

– Supporting the ability to query and modify the data using appropriate query languages,

– Searching the stored information and supporting efficient access to data items,

– Preserving the system coherency, data is scattered in various files of different formats.

• Applications are based on direct access to local file system: they are hard to maintain and to extend.

1- File System Approach

Application

Disk


• A database catalogue is used together with the file system:

– Database provides references to all objects stored locally or remotely

– This approach solves problems related to database overload and improves DB performance

– Data is distributed so that it is physically located closest to intensive usage sites

– Result files can be archived at (or close to) the point where they are generated

– Proper data distribution reduces access bottlenecks at individual sites

2- External Data Link Approach

DatabaseCatalogue

Application

Disk

Link


Two problems are faced when using this approach:

• Database catalogue consistency: referenced binary objects can be updated/removed without notifying the database catalogue maintainer

– Solution: a specific module that automatically and periodically checks the availability of the referenced objects against the database catalogue

• Security issue: referenced objects, which are usually stored in a public location, are not secure.

– Solution: development of a remote file server, through which, the file access and user authentication are controlled based on the database catalogue information

2- External Data Link Approach (Cont.)


• In ideal case, if a database is being created to manage the meta-data, then that database must also store the large scientific data

– This approach solves the problem of keeping the meta-data synchronized with the archive

• However, since the scientific binary data is of huge size:

– It is very costly to store/access the data as large binary objects within the database itself

– Access mechanisms to binary objects require extra encoding/decoding facilities

3- One-Database Storage Solution

Application

Database


4- Parallel/Distributed Database Server

Application

QueryProcessor

StorageManager

TransactionManager

User Queries

Database System

DBMS Software

Repository &Metadata

DB

DB

DB

DB Database Server

• This approach uses a database repository to store the general information, and a distributed database server to store large objects:

– The database repository is better exploited for cataloguing, indexing, and searching facilities

– The database server enforces the issues related to security for access, concurrency control, and information visibility rights.


Parallel/Distributed Database Server Example

DatabaseCatalog

WebInterface

LAN

DatabaseServer

CommonAccess Point

Parallel/distributeddatabase Server

End Users

Web-basedApplication

Remote User

WWWBrowser

DatabaseServer

DatabaseServer

WWWBrowser

StorageManager

Internet

Internet


Distributed Parallel Server Extension

The parallel/distributed database framework provides the MegaStore web server with efficient access to the raw music data. The nodes (music stores) of the distributed MegaStore server are inter-connected, making it possible for specific users to connect to any node in the distributed server and to request an object, without the need to know where that object actually resides

The distributed database supports the following required functionalities:

– Provides a way for managing huge amount of data

– Data is securely kept at geographically distributed music centers

– Data is stored only at the point(s) where it belongs

– Data is visible from any node (music center) within the cooperation community

– Data is efficiently transferred between the nodes in short response time


The MegaStore Prototypes

- The FRS Application

- The LuisterPaal Interface

- Sheet Music Server


Achievement

• Design and development of an e-MegaStore application that seamlessly fits several emerging applications and supports their model of operations

• Design Methodology:– A conceptual model

– A computer system architecture

– An Implementation – Prototype


Conceptual Model

Information Integration and Value

Adding

Content Suppliers Partners

Retailer Shops and home users

Value Adding Partners


System Architecture

Internet

Internet

Additional Inf.

Catalogue

MultimediaDatabase

Major Content Suppliers

FRS NL

FRS BE

Van Leest

Hemi-

sphere

MSI Musi

c

Suppliers

Retail Value Adding

Upload & StorageExternal Links

FireWall

Internet Users

Shop NetworkShop Network

Advanced In-Shop Systems


The FRS-Pilot Application

LAN

Internet Shop

SQL Server

Database Catalogue

Music Content

Back Office

Internet User

WWW Browser

Mu

ltim

edia

Dat

abas

e S

erve

r

Additional Inf.

Catalogue

MultimediaDatabase

Music Center Shops

Catalogue

Cache

Internet

Internet


Security for the Multimedia DB

Short ClipsAudio

Medium TracksAudio/Video

High QualityAudio Tracks

Data is Secure butNot Encrypted

Data is Secure andEncrypted

Data is Secure andHighly Encrypted

Mu

ltim

edia

D

atab

ase

Ser

ver

MultimediaContent


FRS Application Main Interface


FRS Application LuisterPaal Interface


Sheet Music Application Main Interface


Sheet Music Application Subscription Interface


Sheet Music Application Music Book Search Interface


Sheet Music Application Composer Search Interface


Sheet Music Application Music Notes Example


Conclusion

• The main idea behind the developed framework is to design a comprehensive system to support advanced web-applications with two specific characteristics:– to hold large data sets and

– to manage multimedia information

• Thus, the MegaStore system can be considered as a general implementation approach that proves the validity of the proposed architecture and design.

• From this framework, other applications in biology, medicine, and system engineering that share the same characteristics can benefit.


References

• Personnel Home Page:

– http://carol.wins.uva.nl/~ammar

• Cooperative Information Management (CO-IM) Home Page:

– http://carol.wins.uva.nl/~netpeer/

• Matisse DBMS:

– http://www.matisse.com

• Related Publications:– A. Benabdelkader, H. Afsarmanesh, L. O. Hertzberger. MegaStore: Advanced Internet-based

Electronic Commerce Service for Music Industry. In proceedings of 11th IEEE International Conference on Database and Expert Systems Applications - DEXA'2000, Pages 869-878, London - Greenwich, United Kingdom, 2000.

– A. Benabdelkader, H. Afsarmanesh, L. O. Hertzberger. The Virtual MegaStore System Implementation. Technical Report CS-99-05, Faculty of Science, Research Institute Computer Science, University of Amsterdam, 1999.

– A. Benabdelkader, H. Afsarmanesh, L. O. Hertzberger. The Virtual MegaStore System Architecture: Analysis and Design. Technical Report CS-99-04, Faculty of Science, Research Institute Computer Science, University of Amsterdam, 1999.

The MegaStore Application

Documents

Transcript of The MegaStore Application