graduate thesis - User Interface Research Group - IGARASHI

µ-SCRIPT-SYSTEM : A FRAMEWORK OF

SYNCHRONIZED MUSIC CONTENTS USING A SCRIPT

LANGUAGE

µ-Script-System : スクリプト言語を用いた同期機能を持つ音

楽コンテンツのためのフレームワーク

by

Hidehiko Abe

阿部秀彦

A Senior Thesis

卒業論文

Submitted to

the Department of Information Science

the Faculty of Science, the University of Tokyo

on February 1, 2005

in Partial Fulfillment of the Requirementsfor the Degree of Bachelor of Science

Thesis Supervisor: Takeo Igarashi 五十嵐健夫assistant professor of Information Science

ABSTRACT

This paper proposes a new framework in which events described in a script language are embedded in

Music XML. As the Internet becomes popular and the network technology improves, interactive musical

contents rapidly and widely spread. In making musical content, it is important to synchronize other

elements such as animations and user interaction. When we make such contents, however, we usually

control timings based on general time units, like millisecond or microsecond, but not on rhythmical ones,

like measure, beat or tick. This paper presents a new framework in which the control information can be

described inside the musical information, which has been conventionally described separately, by using

an easy script language. This makes the timing control of events more flexible and intuitive. We chose

Music XML as a format of the musical information. Music XML is a text-based new musical format.

The extended Music XML, in which the script is embedded, is not a format intended for exclusive use

with µ-Script System, but can be treated as usual Music XML by ignoring the script tags. Furthermore,

this system meets various demands of users by the interaction with external applications. This would

facilitate the creation of music contents.

論文要旨

本稿は Music XMLに対応したイベントをスクリプト言語で埋め込む新しい枠組みについて提

案する。インターネットの普及、ネットワーク技術の向上によってインタラクティブな音楽コンテ

ンツが急速にひろまり、より身近なものになってきている。音楽コンテンツを作るうえで、曲の進

行と、例えば動画の表示やユーザインタラクションといったその他の要素との間の同期は大切で

ある。しかし現在、そのようなコンテンツを作るときの同期の取り方は、ミリ秒やマイクロ秒と

いった一般的な時間単位をベースとしたタイミング制御に基づくものであり、音楽の時間単位の

基本である拍子や小節に合わせたタイミングの制御を行うような機構はないように思われる。本

稿では、従来別々に記述していた情報を音楽情報中に簡単なスクリプト言語で記述できるような

枠組みを提案する。これによりイベントのタイミング制御をより柔軟かつ自然に行うことができ

る。また、スクリプトを埋め込む音楽情報として、本研究ではMusic XMLを用いることとした。

Music XMLは新しいテキストベースの音楽フォーマットである。本稿で扱うフォーマットはシス

テム専用のものではなく、埋め込んだスクリプトタグを読み捨てることで、通常のMusic XMLと

しても扱える。さらに、本システムは外のアプリケーションと連携をとることで、よりいろいろ

なユーザの要求にこたえられる。これによって音楽コンテンツの作成を支援できるようになる。

Acknowledgements

We thank assistant professor Takeo Igarashi for his great advice. This work would not be

completed without his help. We are also grateful to members of Igarashi group, especially Mr.

Shigeru Owada and Mr. Yasushi Maruyama, for their comments on our work. Many ideas in

this work stemmed from discussion with them.

Contents

1 Introduction 1

2 µ-Script System 3

3 Design 5

3.1 Music data format . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

3.2 Music XML Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

3.3 Script Tags . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

3.4 µ-script language . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

3.4.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

3.4.2 Values . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

3.4.3 Statements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

3.4.4 I/O and Connections . . . . . . . . . . . . . . . . . . . . . . . . . . 10

3.4.5 APIs (Application Programming Interfaces) . . . . . . . . . . . . . . 10

4 Implementation 15

4.1 Reading Music XML file . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

4.2 Conversion into MIDI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

4.2.1 Calculation of resolution . . . . . . . . . . . . . . . . . . . . . . . . 15

4.2.2 Calculation of ticks at the beginning of each bar . . . . . . . . . . . . 16

4.2.3 Conversion from note tags to MIDI event . . . . . . . . . . . . . . . 17

4.3 Interpretation of µ-Script . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

4.4 Interpreter Behavior . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

4.5 An Example Application . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

5 Result 21

6 Discussion 25

i

7 Related work 26

8 Conclusion 27

A Syntax of µ-Script Language 28

A.1 Lexical Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

A.1.1 Keywords . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

A.1.2 Punctors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

A.1.3 Numbers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

A.1.4 Comments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

A.1.5 Strings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

A.1.6 Identifiers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

A.2 Syntax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

References 33

ii

List of Figures

2.1 Image overview of coventional system. . . . . . . . . . . . . . . . . . . . . . 3

2.2 Image overview of µ-Script System . . . . . . . . . . . . . . . . . . . . . . 3

2.3 The image of a file treated µ-Script System . . . . . . . . . . . . . . . . . . 4

4.1 GUI of the implemented interpreter . . . . . . . . . . . . . . . . . . . . . . 19

5.1 The tempo controller by socket connecting . . . . . . . . . . . . . . . . . . . 24

iii

Chapter 1

Introduction

Today, music plays an important role in interesting and interactive digital contents. Two major

reasons are the following. First, the rapid development of the Internet technology enabled a

large amount of data to be sent and received. Second, various kind of data can be processed

in short time thanks to higher spec CPU.

Though famous applications like Macromedia Flash [9] are popular in creating digital

contents that includes musical element, they have a problem in that synchronization is not

realized between musical and unmusical elements in those applications. The music elements

synchronize with unmusical ones only at the moment when the sound begins, and no systems

keep synchronization between musical time lines and unmusical ones after that. When the ini-

tialization fails or these time lines get disaccorded, the gap can not be modified. The second

problem is that the time units used in those applications is not as same as the ones used in

music. The former units are millisecond, microsecond, or second and the latter ones are mea-

sure, beat, or tick – a general time unit in MIDI (Musical Instrument Digital Interface) [10].

This will be inconvenient when, for example, creators want to make contents in which a robot

dances to the music. They need to convert the musical time unit into the general time unit

by means of a special calculation. The performance of the result of the calculation, however,

depends on the execution environment. Therefore, they need to repeat tests and modifications

to get the best correspondence.

This paper proposes a new framework that is based on musical time units and constant

synchronization while the music is going on. As a result, it will be possible to play music

in perfect concordance. It will be easy for the user to execute an event at any bar, any beats

they want. For example, users are enabled to set the number of repetitions, branches of music,

conditions and connections to servers. This system can be connected to external applications

by sockets. This would faciliate the creation of various contents. For example, when a creater

1

requires treating an external device which cannot be connected directly to this system, a simple

application which treats both the device and a socket enables to make a connection between

the device and the system.

Section 2 discusses a overview of µ-Script System. Section 3 presents the design of µ-

Script Language and the way to embed the script in the Music XML. Section 4 presents an

implementation of this system. In Section 5, a small and simple sample is shown. Section

6 discusses this system in the present version of implementation. In Section 7, we examine

related work. Finally, the summary of the conclusions in Section 8.

2

Chapter 2

µ-Script System

The simple way to treat music in conventional systems is to send music data to a music player

and order it to play the music (see figure 2.1). The music player can be both hardware and

software. A music file bears little relation to even remotely related to unmusical files in an ap-

plication. Additionaly, the data are sent only from an application to a music player. Therefore,

music seems to be isolated in the system.

µ-Script System is a model of connecting unmusical application with a music player (see

figure 2.2). A file treated in this system includes both music and unmusical data (see figuare

2.3). Therefore, it is easy for a music creater to set timings of events which is related to

music. This system connects unmusical data to music more strongly and makes unmusical

data synchronize with music.

Figure 2.1: Image overview of coven-

tional system.

Figure 2.2: Image overview of µ-

Script System

This system is designed to create interactive contents based on music, for example, a danc-

ing robot. Progress of music determines the timing of actions of the robot, such as waving

3

Figure 2.3: The image of a file treated µ-Script System

its hands and executing a turn. The robot looks its best on a stage with backlights blinking in

synchronization with the music. The stage might have fountains shape-shifting in the synchro-

nization. Therefore, since music is a base of the actions and the blinking and shape-shifting

patterns, these events should be described in music. Moreover, when interactions are added,

synchronization of this system will make a significant contribution. Since this system keeps

dynamically synchronization differently from conventional systems, gaps will be modified.

Therefore this system may suitable to such contents.

4

Chapter 3

Design

3.1 Music data format

This paper proposes a new simple script language, µ-Script Language, to write unmusical

information, in music data. Although most of the music data formats are binary ones, such as

SMF (Standard MIDI-file Format) [10] or Wav, MP3 etc, textual data format is more suitable

to describe a simple script language inside the music data. MML (Music Macro Language),

which has some specifications, is a famous textual data format, but it got out dated, so we

choose Music XML [4, 5, 3] as a music file format to describe µ-Script Language. Music

XML is a very new format, the version 1.0 of which was announced in January, 2004. Many

famous applications in the field of treating digital music, like Finale [6], come to support Mu-

sic XML file format. Moreover, it is a data format suitable to be distributed in the Internet, and

it matches to the purpose of this study of easily making and distributing music synchronized

contents.

3.2 Music XML Overview

This section presents an outline of Music XML which is used in this thesis. Music XML

data structure is roughly divided into four parts. In the first part, only music notation infor-

mation is described; what clef and what rhythm sign is to be chosen, how slurs, ties and acci-

dentals are to be described, where bar-lines are to be described, or which stems, note heads,

beams are chosen? In the second part, only music performance information is described. This

is the information needed to play music; how high or low the real pitch level is in comparison

with the step in chromatic scale, how long the note duration is, or whether the note is tied

or not. The third part includes both music notation and music performance information, like

steps. A step element indicates both where the note head comes and how high the note pitch

5

is. The last part is a set of unmusical information, like the name of the composer, the title,

score groups, etc.

Music XML has two types. One is ”score-partwise”, which is a series of parts with mea-

sures inside. The other is ”score-timewise”, which is a series of measures with parts inside.

Only score-partwise is supported in this paper since there was especially no essential differ-

ence between them. We will here after treat only ”score-partwise”, if not particularly men-

tioned.

This paper focuses on the performance part of Music XML. There are many tags in Music

XML, but only some main tags are paid attention to in this paper. They are roughly explained

in the following. When details are needed, they will be supplemented.

• alter tag: shows how high or low the note is in chromatic scale.

• attributes tag: is used to set and change the new attribute from old ones.

• backup tag: means to return back for the specified duration from current position.

• divisions tag: is put in attributes tags. It means to change resolution from current posi-

tion. Resolution is the minimum unit.

• duration tag: means the length of the note or the rest.

• forward tag: means to go ahead for the specified duration from the current position.

• measure tag: is a unit of measure.

• note tag: is a unit of note or rest.

• octave tag: specifies the octave where the note is described.

• part tag: means one of the song parts. It has measure tags as children in the score-

partwise type.

• part-list tag: declares each part.

• part-name tag: declares the name of the part.

• pitch tag: assigns the pitch. This tag includes an octave tag and a step tag as its children,

and also is possible to include a alter tag. These children specify musical performance

pitch. It has a note tag as its parent. A note tag which has a pitch tag as its child will

signify a note, not rest.

• rest tag: The parent note tag which has rest tag signifies rest.

6

• score-part tag: declares a part.

• score-partwise tag: is a root of Music XML in the score-partwise type.

• step tag: includes any one letter of ’C’, ’D’, ’E’, ’F’, ’G’, A’, or ’B’, which respectively

represent the sound of do, re, mi, and so on. This element means the step of the grand-

parent node, namely note.

• tie tag: This tag is used in pair. One of the pair has the start attribute and the other has

the stop attribute. Their parent note tags should have the same pitch.

3.3 Script Tags

µ-script language is embedded in <script>·· ·</script> tags in the Music XML data.

It controls the execution timing by the position where it is described. There are two kinds of

the execution timing. One is the initialization at the beginning and the other is done during

musical performance with synchronization. The processes which may be a burden afterwards

during the synchronization should be done before the music starts. It includes such tasks as

creation and connection of socket, declaration of global variables, and to set handlers of input

queues or sockets. These tasks will be explained in detail in chapter 4.

To achieve these two types of execution, script tags are allowed to be written both in score-

partwise tags and attributes tags. The script written in score-partwise tags is interpreted as the

initialization when this system reads a Music XML file, since score-partwise tags should be

unique in a tune.

The other is for the events during performance. These scripts to go are allowed to be

written directly under the attributes tags, for the reason following. The row of duration tags

shows the flow of time in Music XML. Notes and rests have duration. A note is played

just after the end of the previous note, rest, etc. Since these notes and rests can only be

monophonic. Forward tags and backup tags solve this problem. The former tags set ahead the

current timing, and the latter push it back, with duration. Measures tag can have all these tags.

Measures tag can also have attributes tags. Attributes tags are written between those tags and

have no duration. Therefore attributes tag signifies a particular point of time in Music XML.

That is why script tags should be written in attributes tags.

7

3.4 µ-script language

3.4.1 Overview

µ-script language is a simple script language which is embedded in Music XML files.

Syntax of theμ-script language is almost equivalent to the Subset of JavaScript (see Appendix

A) [11].

3.4.2 Values

The value types in µ-script language are as follows:

• Signed integer of 64 bits

• Unsigned integer of 64 bits

• Floating point number of 64 bits and 32 bits

• Boolean values

• String

• Arrays

• Sockets

• Null

• Undefined

• Functions

64 bits floating point number is used as basic numerical value. Constant numerical values

are recognized as this. If users want to use other types of numerical value, it needs to add

a suffix, which is ’f’ or ’F’ for 32 bits floating point numbers, ’l’ or ’L’ for 64 bits signed

integers, and ’u’ or ’U’ for 64 bits unsigned integers.

A character is one byte, and String is an array of bytes. Constant string values are written

in the form of ”. . .” or . . .’. The difference between them is the terminal symbol. Strings

starting with a double quotation mark end with the same mark. Single quotation marks are

recognized as normal characters. On the other hand, those starting with single quotation mark

end with the same mark, too. Double quotation marks are recognized as normal characters. To

use single quotations in ’. . .’ or double quotations in ”. . .”, it is necessary for these characters

to be escaped by a special character \. BS, LF, CR, and TAB can be respectively represented

8

by the escaped b, n, r, and t. It is possible to describe the byte value directly in the form of

\ooo or \xhh. An o character is an octal number character and an h character is a hexadecimal

number character. These values must be positive and less than 25610.

Null value is used to clear handles or to set non-reference value. Undefined value is a

result of evaluating uninitialized identifiers or functions that return a void, etc.

Function type is a simple function. No closures are generated.

Sockets are generated by the function ”Socket”. This function has two parameters; the

first is a host name, the second is a port number. When it fails to connect the server, the value

of the Socket type is not generated.

Array values in µ-script language are fixed length array. No systems identify the value

types in this language. Therefore any values can be set in an array. The Boolean value, for

example, can be set as an element of array which is a function value before substitution. The

same is true for variables. Moreover, various types can be mixed in an array. For example,

the first element can be a socket in a three elements array, while the second is a Boolean value

and the third is a function. A multi-dimensional array is expressed by an array which contains

arrays.

All values can be evaluated as Boolean. Boolean entity is true or false. Arrays, functions,

and sockets are always true. Null and undefined are always false. The numerical value is false

when it is 0, NaN, Positive Infinity, or Negative Infinity; otherwise the value will be true. Zero

length strings are false and other strings are true.

3.4.3 Statements

Statements should be separated by semicolon characters. Semicolons of a last statement-

list can be omitted, according to the context.

When the script interpreter fails to evaluate an expression in a statement, it stops interpret-

ing the statement, then restart the interpretation by the following statement.

This language has four statements to control the programming flow. These are ’for’,

’while’, ’do-while’ and ’if’ statement. ’Continue’ and ’break’ statements can be used

in ’for’, ’while’ and ’do-while’ statements. ’Return’ statements are allowed to be used

anywhere in the function body.

This system is able to recognize common fundamental floating point arithmetic operators,

namely, +, -, *, /, %, <, >, <=, >=, ==, and !=. It seems uncommon that it is recognizes bitwise

operators, such as, ~, &, |, ^, <<, >> and >>>. In using bitwise operators, the interpreter

converts a number to an integer before the operation, and it again converts the result converts

to the original numerical type after the operation. All of the Boolean operators, namely, & &,

9

|| and ^^, can be applied to all values, since all types of values can be evaluated as Boolean

ones. Besides, this language has a substitution operator, which is =, and complex substitution

operators, that is, +=, -=, *=, /=, %=, ~=, & =, |=, =̂, <<=, >>=, >>>=, & & =, ||= and ^^=.

This script also has block statements. The scope of the identifiers declared in a block

statement is limited to the block.

Function application fails when the number of parameters of the function is not equal to the

number of actually given arguments. Otherwise, the expressions as arguments are evaluated.

Next, the interpreter creates a new environment that shares only global environment with

others, where the corresponding parameters refer the evaluated values. Then, the function

body statements get evaluated. Finally, the created environment is destroyed and the results

are returned.

3.4.4 I/O and Connections

This language has two input and two output mechanisms. Socket connection paly the role

of both input and output. This system is controlled by three primitive functions. These are

”Socket”, ”Recv”, and ”Send”. Each socket value has two byte-wise queues. One of them is

a buffer to send data to the server and the other is to receive from the server. When ”Socket”

is called, the interpreter creates a native socket and tries to connect the server specified by

the arguments. When this process succeeds, it creates a socket value of µ-script system and

creates a thread that polls the native socket and delivers the data from the native socket to

buffer and vice versa in the background. We adapt such a complex system for fear that the

connections interrupt the music.

Message queue is the other type of input. Any value can be added to this queue, such

as arrays, number, or string. When values are added, a handler is called with an input value

as an argument. Stream is the other type of output. This will be in use when ”PrintInt”,

”PrintDouble” or ”PrintString” function is called.

3.4.5 APIs (Application Programming Interfaces)

µ-script language has primitive functions as follows:

� CreateArray(size) :

This function generates a fixed length array, in which the number of elements is ’size’.

If size is not a number or is negative number, it will be an error and the evaluation stops.

All elements in generated array are Undefined.

� size: Number of elements of generated array.

10

� Return value: An array whose element size is ’size’.

� ArrayToString(array) :

This function converts an array of the numerical values into character string. The length

of the result is equal to the input array length. If there exists one or more value that is

not a numerical value in the input element, it will be an error. To convert a number to a

byte, each of number is cast to a byte modulo by 256.

� array: Number’s array to be converted.

� Return value: a converted string.

� StringToArray(str) : This function converts a string into an array of the numerical

value.

� str: a string to be converted.

� Return value: a converted array of numerical values.

� GetLength(in) :

This function returns the length of an input array or string.

� in: a string or an array whose length a user wants to know.

� Return value: Length

� GetResolution() :

This function returns the resolution of a tune installed in an interpreter.

� Return value: Resolution.

� PrintInt(num) :

This function converts the input value into 32 bits signed integer and output it to a

stream.

� num: A number to output to stream.

� Return value: Undefined.

� PrintDouble(num) :

This function converts the input value into 64 bits floating point number and output it to

a stream.

� num: A number to output to stream.


11

� PrintString(str) :

This outputs the input string. ’\ 0’ character is recognized as a terminal symbol in a

similar way of C language.

� str: A string to output to stream

� Return value: Undefined

� Jump(ticks) :

This function sets the current time to the input position. Current statements get exe-

cuted. If a script is described in the newly fixed time, it is also executed.

� ticks: The destination position with tick unit.


� Stop() :

This stops the performance.


� Transpose(num) :

This sets a transpose as an absolute value. To set transpose as a relative value, a function

defined below is needed.

_transpose = 0;

function RelativeTranspose(num){

_transpose += num;

Transpose(_transepose);

}

� num: A number of transpose.


� SetHandler(func) :

This function set ’func’ as a handler of an input queue. When the queue gets input

message, the handler is called with the message as argument. To clear the handler, this

function is called with ’func’ being null.

� func: The handler function of input queue.


12

� ToTick(measure, beat, ticks ) :

This converts input measure, beat, and ticks into MIDI ticks.

� measure: Measure number to be converted.

� beat: Beat number to be converted.

� ticks: Tick number to be converted.

� Return value: Converted ticks.

� GetTicks() :

Current position is returned.

� Return value: Current position.

� Socket(hostname, port ) :

This function creates a socket connected to the server specified with arguments. When

it fails to connect to the server, it reports an error.

� hostname: A string that specify a server.

� port: The port number.

� Return value: A socket value connected to the server.

� Recv(socket, length) :

This reads data from the socket up to the ’length’ bytes.

� socket: A socket to read.

� length: The maximum length of to read data.

� Return value: A string buffering read data.

� Send(socket, msg ):

This writes data in the socket.

� socket: A socket to write data in.

� msg: A string buffering data to send.


� SetTempo(bpm) :

This sets tempo.

� bpm: A number of beats per minute.


13

� SetSocketHandler(sock, handle) :

The function is called to set a socket handler. The handler is called when data are in the

socket. If buffered data are not read in the handler, the handler continues to be called

with sock as argument.

� sock: A corresponded socket.

� handle: A function of handler.


14

Chapter 4

Implementation

4.1 Reading Music XML file

This system uses ”expat”, one of the most famous and fastest libraries in the world of

parsing XML files of non-verification type. The present specification of this system disregards

unnecessary part when converting it into MIDI data. Tags which can be read in Music XML

at present are alter tags, attributes tags, divisions tags, duration tags, measure tags, note tags,

octave tags, part tags, part-list tags, part-name tags, pitch tags, rest tags, score-part tags, score-

partwise tags, step tags, and voice tags. Script tags can be added to this list of tags. The

script language can be described between the beginning script tag and the ending one. A

problem is caused here. µ-script language has less symbols ,’<’ and ’<=’, but the symbol is

recognized as beginning of tags in expat, and parsing fails. There are two ways to avoid this

problem. One is to use greater symbols, ’>’ and ’>=’, instead of the less symbols, since the ’>’

character is recognized as a normal character in expat library. The other is to describe scripts

in ”<![CDATA[.. . ]]>” tags. All of the characters are yet to be recognized as script in this

phase.

4.2 Conversion into MIDI

Conversion from Music XML to MIDI is divided in three phases, calculation of resolution

of the whole music, calculation of ticks at the beginning of each bars, and conversion from

note tags to MIDI event.

4.2.1 Calculation of resolution

Resolution is a unit of the time stamp in MIDI. It is easy to calculate the resolution. Music

XML expresses time by duration. This unit of duration is specified by the divisions of the

15

Attributes tag. Divisions are the index of how many equal parts there are in a given musical

time units. Therefore, resolution is the least common multiple of all divisions of music, which

is then multiplied by ten. This multiplation is indispensable, because the durations needs to

be an integer after multiplied by 0.9. The detail is explained in 4.2.3

4.2.2 Calculation of ticks at the beginning of each bar

Good showed a simple algorithm to calculate length of duration [2]. The time stamp of

each bar is determined by the duration included in the bar. Each bar has elements which have

their children of note, backup, and forward. After searching for durations in all parts, the

length of a measure of each part is calculated. The measure length is the maximum value of

those of each part. However, all the lengths of a bar of each part should be equal in general.

Although there are some rare cases in which they are not equal, for example, when time

signatures of each part are not equal, this system doesn’t support those cases. The length of a

note and a rest is determined by the current divisions and its duration, shown as follows:

ticks =resolution×duration

current divisions

Current divisions is the value of divisions in the attributes tag described immediately be-

fore. Since resolution is the least common multiples of the sets of current divisions multiplied

by ten, as previously shown, note duration is also an integer. The length of a bar of a part,

mp, is equal to the summation of these lengths subtracted by the sum of backup durations as

follows:

lengthmp = ∑note\chord∈mp

ticksnote + ∑forward∈mp

ticksforward − ∑backup∈mp

ticksbackup

The measure length is the maximum value of the length of a measure of each part.

lengthm = maxp

{lengthmp

}

At last, the timestamp of the beginning of measure is summation of all of the preceding

measure lengths.

timestampm = ∑m′precedem

lengthm′

16

4.2.3 Conversion from note tags to MIDI event

First, it is necessary to decide the height of the note. Note number, which is used as pitch

in MIDI, is calculated from the step, the octave, and the alter of a note as follows:

notenumber = (octave+ 2)×12 + alter +

0 if step = ’C’

2 if step = ’D’

4 if step = ’E’

5 if step = ’F’

7 if step = ’G’

9 if step = ’A’

11 if step = ’B’

If the alter is not defined, it will be zero. The length of each note is defined as follows:

notelength =resolution×duration

current division×0.9

Current divisions is the same as was above. The reason of the multiplation 0.9 is that a

note is not performed at the whole the length when actually performing. Since resolution is

multiplied by ten, note length is integer. In MIDI, a note is represented by thw events. One

is ”note on” event, which is message 0x90{note number}{note velocity}. The other is ”note

off” event, which is 0x80{note number}00. 0x90 or 0x80 are status bytes. The note on event

is sent at the beginning of the note, and the note off event is after the time passed for the length

of the note. In this paper, note velocity is constantly 0x5A. The current time advances by the

note ticks, not by the note length.

In the case of a note with tie, note off event of the note with starting tie is not sent, and

note on event of the note with stopping tie is not sent. If a note has both the starting tie and

the stopping tie, both events are ignored. Moreover, when a note which has a chord tag is

converted to events, the timestamp of the note on event will be as same as the previous note,

and that of the note off event is recalculated from the duration of that note. The chord note

does not change the current time. When the forward tag, which is not converted to MIDI

events, puts the current time forward for the ticks calculated from the duration of note. The

backup tag is not converted to MIDI events neither, and it puts back the current position for

the ticks calculated from the duration of note.

When a script tag is found, the lexical analusis and the parsing of the script language

occurs. If the parsing is succeed, the result can be found in the current position. The current

position does not move.

17

4.3 Interpretation of µ-Script

µ-Script system uses Bison and Flex to make its parser and lexical analyzer. For the actual

grammar, see Appendix A. This script language interpreter has GC (Garbage Collector). GC

uses famous algorithm, Mark and Sweep. It controlls all of the expressions, statements, and

values. GC is executed before a new XML file is read in the present version of implementation.

A synchronized queue as a message queue is the standard input. The array is added to this

queue. When there is input data, the interpreter passes the data to handler as an argument,

and the handler is executed. When handler is not set or the number of the parameter of the

handler is not equal to the length of the input data, the data is immediately ignored. On the

other hand, the output is C++ Stream. The stream is connected with the interpreter, and data

from the interpreter is received.

One of the main features of this system is a communication of ”Socket”. When ”Socket”

is created, it tries to connect the server host and port, which are taken as arguments. When

the connection fails, it will be an error. Otherwise, the thread is created for each socket and

watches the socket, while two thread-safe byte data queues are created. One of the queues is

used to buffer the received data. The other is used to buffer the data to be sent. When ”Send”

is called, the data is copied in a buffered queue. Then the flow immediately returns to the point

where the function was called.

4.4 Interpreter Behavior

A fast thread is generated at the beginning of the interpretation to control multimedia

without delay. Execution is done in the thread in the following order. First, the present timing

is calculated. The timing is calculated from Tempo and resolution as follows:

time[ticks] = passedtime[msec]× resolution[ticks/beat]×bpm[beat/sec]×60×1000

The passed time is the diffrence between the time of last phase and the current time. The

present position is the total of the passed-time ticks times.

Next, input message queue is checked. When data come, only a piece of data is read

per execution of thread. It is passed to a handler function as an argument and the function

is execution if the handler is set and the number of parameters is equal to the number of the

arguments. Otherwise, the input data is thrown away. The purpose of this restriction is not to

disturb the performance of music when a large amount of data suddenly comes. Input data is

thrown away when the handle is not set.

18

Then, it is checked whether handlers of sockets are set or not. When a handler is set in

socket, and when the buffer of socket has data, the handler is called with the socket as the

argument. Each socket is checked only once. At this moment, if the handler does not read out

the data, the data remains in the buffer, which makes the handler to be read again.

When all the call checks of handler end, the search and execution of a script is done if

there exist any scripts that should be executed in present time or the time immediately before.

When API of the script movement is called, MIDI Player assumes that time has passed while

ongoing the script is executed until last minute. The interpreter searches for scripts that exists

exactly at the same time stamp. If there is any, the script will be executed.

After processing all the scripts upto the end, the MIDI message is transmitted.

4.5 An Example Application

The figure 4.1 is the GUI of an example application.

Figure 4.1: GUI of the implemented interpreter

The ”Open” button is used to find an xml file and to open the file. When the file opens,

this system reads the data. First, it is parsed and is converted to MIDI data. Next, µ-Script

language is lexcally analyzed and parsed. Finally, the script for initialization is executed.

19

Clicking ”Play” button begins the performance and the script is executed at the given

timing. The ”Stop” button stops the performance. The text box at the upper left side is a

window for a simple experiment of script execution. When a script is written here and the

”Exec” button is clicked, the script is parsed and interpreted. Although the interpretation

has no effects on the environments which is used in performance at the moment, it effects to

outside devices, such as a music player or MIDI device.

The window at the upper right side is a text output window which is connected to a stan-

dard output stream of the interpreter. If ”PrintInt”, ”PrintDouble” or ”PrintString” function is

called, the data from the interpreter is displayed in the window.

20

Chapter 5

Result

A small and simple sample is shown. It is a fragment from the famous tune of a video game

Super Mario Brothers. The script embedded the sample controls the tempo and the flow of the

music.

the sample is as follows:

<?xml version="1.0" standalone="no"?><!DOCTYPE score-partwise PUBLIC

"-//Recordare//DTD MusicXML 1.0 Partwise//EN""http://www.musicxml.org/dtds/partwise.dtd">

<score-partwise><part-list>

<score-part id="P1"><part-name>Voice1</part-name>

</score-part><score-part id="P2">

<part-name>Voice2</part-name></score-part><score-part id="P3">

<part-name>Voice3</part-name></score-part>

</part-list><script><![CDATA[

sock7 = Socket("127.0.0.1", 10006);function sockhandler (sock){

str = Recv(sock, 4);array = StringToArray(str);tmp = array[0];if( 0 > tmp ){

tmp += 256;}SetTempo(tmp);

};ret = 0;function handler(){

ret = GetTicks();Jump(ToTick(38,1,0));SetHandler(null);

};SetHandler(handler);

21

SetSocketHandler(sock7, sockhandler);]]></script><part id="P1">

<measure number="1"><attributes>

<divisions>4</divisions><clef>

<sign>G</sign><line>2</line>

</clef></attributes><note>

<pitch><step>E</step><octave>4</octave>

</pitch><duration>1</duration>

</note><note>



</note><note>

<rest /><duration>1</duration>

</note><note>



</note>::<measure number="38">

<attributes><divisions>12</divisions>

</attributes><note>

<pitch><step>C</step><octave>4</octave>


</note><note>

<pitch><step>G</step><octave>3</octave>


</note><note>

<pitch><step>C</step>

22

<octave>4</octave></pitch><duration>1</duration>

</note><note>



</note>::

<note><pitch>

<step>B</step><octave>4</octave><alter>-1</alter>


</note><note>

<pitch><step>F</step><octave>4</octave>


</note><note>

<pitch><step>B</step><octave>4</octave><alter>-1</alter>


</note><note>

<pitch><step>D</step><octave>5</octave>


</note><note>



</note><note>

<pitch><step>B</step><octave>5</octave><alter>-1</alter>


</note><note>

23



</note><attributes>

<script>Jump(ret);ret = 0;SetHandler(handler);

</script></attributes>

::

Two scripts are embedded in the sample. The first script is described in score-partwise

tags. It is are at the moment of initialization. It shows that the interpreter creates a socket

and then sets each handler to the socket and to the message queue, a standard input. The

socket handler needs a byte from the socket and sets it as beat per minute. In the message

queue handler, the interpreter stores the current position, then set 38th bar as playing position.

A sound effect played when Mario gets a Super Mushroom is described at this bar and the

interpreter does not accept the standard input while the sound is playing. It returns to the

stored position when the sound effect ends, and it comes to accept the input again.

Additionally, a simple Java application is implemented which is connected to the inter-

preter and controlls tempo.(see figure 5.1)

Figure 5.1: The tempo controller by socket connecting

This application is connected to the interpreter with a socket. When the value of its slide

bar changes, it sends a byte which is equal to the scale of the slider through the socket.

The current tempo can be controled by using the slider from the perspective of a user.

When the slider is moved to the left side, the tempo slows down. When the slider is moved

to the right side, the tempo quickens up. Additionaly, when the ”GO!(Message)” button (see

figure 4.1) is cliked, the sound effect interrupts the main music.

24

Chapter 6

Discussion

µ-Script System in the present version of implementation is able to treat simple and small

samples, such as Mario. It is able to read script tags and a subset of Music XML tags and

converts from them to MIDI events. µ-Script Language is lexical analyzed and parsed by this

system. The system controls a flow of music, transposition, and a tempo by calling APIs of

the language. Moreover, socket connections are allowed in this system. They are powerful in

coordination with external applications and devices. The tempo is controled by using both a

socket and APIs in the Mario sample.

However, the system has three problems in the present version of implementation. The

first one is that it is difficult to describe complex scripts since the syntax of µ-Script Language

is small. Additionally, a kind of the API is not enough. It is necessary to extends the syntax

and APIs, so as to support class and closure or to enable to control a window. The second is

that the system cannot read all of the Music XML tags. Some unread tags should be converted

to MIDI events. Hence, the system plays only the piano, for example. It is necessary to

implement both a reader and a converter which treat these tags. The thrid is the GC is not

efficient. When this system runs for a long stretch of time, unnecessary objects might increase

and burden the software. They might prevent this system running at worst. It is necessary to

improve in the quality of GC.

25

Chapter 7

Related work

CAL (Cakewalk Application Language) [7] is a language that treats music. It is similar to Lisp

in that it is a function programming language. It is not appropriate to treat music, because

music is a proceedure, not a function. In this language, prefix operators are used, but infix

ones are not. For instance, ”(+ 1 1)” in Lisp means ”1 + 1” in mathematics, and ”(< x

y)” means ”x < y”. Additionaly, a pair of parentheses is a function application operator. It

makes easy to lexically analize and parse the language. Since function applications are nested,

however, there are too many parentheses and it is not friendly for users when the nest is deep.

Macromedia Flash is a famous application in creating contents which synchronize with

music. Music is treated as background music in this appliction. Therefore users set only the

beginning of music, but can not control it afterwards. Since files used in Flash are binary

format, it is difficult for the famous internet searching systems to find and make an index of

these files and the development kits is needed to create contents.

Music XML is fast growing new file format in digital music [1]. This format can be used

in many famous application, such as Finale or Sibelius [8].

Script language is embedded to many other languages, such as HTML, VRML97(Virtual

Reality Modeling Language) [12] and Maya [13].

Java Script is found most often in HTML files. It gives movement to objects, such as text

data, images and browsers and can also get and handle user interactions. VRML97 is a format

to describe interactive 3D objects for world wide web. Java Script customized for the format

can be used. It is written in a script node. It handles events, such as user interactions, and

controls objects and environments of VRML. Maya is a complete workflow solution of mod-

eling, animation, visual effects, and advanced rendering technologies. MEL (Maya Embedded

Language) can be used in Maya. It is such a language as Java mixing with a shell script. It

enables to customize user interfaces and to create original visual effects.

26

Chapter 8

Conclusion

This paper have proposed the design and the implementation of the µ-Script System. This

system treats Music XML files embedded the µ-Script Language. It makes unmusical ele-

ments synchronize with music during the performance of the music. Therefore, it repaires the

unexpected gap between music and unmusical elements which may be caused depending on

performance environment.

In chapter 2, this paper have proposed a framework of µ-Script System. This system

model is contrasted with conventional one. Then it have been shown that the system may be

suitable to contents based on and synchronized with music.

In chapter 3, µ-Script Language have been designed. It has a similar syntax to Java Script

and has APIs which controls a music player. Additionaly, it has a message queue and sockets.

Therefore, it connects to external applications, such as a tempo control sample application.

In chapter 4, µ-Script Interpreter have been implemented. It has tree functions; The first

one is a reader of Music XML files in which µ-Script is embedded. The second is a converter

from Music XML data to MIDI events. The third is the main interpreter of µ-Script. This has

a fast thread to execute main tasks and background threads to connect to servers through each

socket.

In chapter 5, a small and simple sample, a background music of Mario, have been shown

and have tried running on this system. It controls the tempo and the flow of music.

In future, we would like to make a program library of this system and would like to dis-

tribute it. Since it treats large files, it will be necessary to develop an authoring tool for this

system. Finally, it is necessary to create applications which enable more complex events, for

example, operating phisical objects.

27

Appendix A

Syntax of µ-Script Language

A.1 Lexical Syntax

A.1.1 Keywords

The keywords of this language are shown as follows:

abstract, as, break, case, catch, class, const, continue, debugger,

default, delete, do, else, enum, export, extends, false, finally, for,

function, get, goto, if, implements, import, in, include, instanceof,

interface, is, namespace, native, new, null, package, private, protected,

public, return, set, super, switch, synchronized, this, throw, throws,

transient, true, try, typeof, use, var, volatile, void, while, with,

However, there are keywords which are not implemented.

A.1.2 Punctors

The punctors of this language are shown as follows:

!, !=, !==, %, %=, &, &&, &&=, &=, (, ), *, *=, +, ++, +=, ,, -, --, -=, ., ..., /,

/=, :, ::, ;, <, <<, <<=, <=, =, ==, ===, >, >=, >>, >>=, >>>, >>>=, ?, [, ], ^, ^=,

^^, ^^=, {, |, |=, ||, ||=, }, ~

A.1.3 Numbers

The number of this language are shown as follows:

• HI : 0 [xX] [0-9A-Fa-f]+

• DI : 0 | ( [1-9] [0-9]*)

28

• FRAC : [0-9]*

• MANT : (DI (. FRAC?)?) | (.FRAC)

• EP : [eE] [+-]? [0-9]+

• 64bits floating-point number : HI | (MANT EP?)

• 32bits floating-point number : MANT EP? [fF]

• 64bits signed integer : (DI | HI) [lL]

• 64bits unsigned integer : (DI | HI) [uU] [lL]

A.1.4 Comments

The comments are the form of /* . . . */, and one line comments are the form of // . . . ↵

A.1.5 Strings

The strings are the form of " . . . " or ’ . . . ’.

A.1.6 Identifiers

The identifiers are shown as follows:

• identifier : [A-Za-z ] [A-Za-z 0-9]*

A.2 Syntax

simple.qualified.identifier : identifier

qualified.identifier : simple.qualified.identifier

primary.expression : null | true | false | NUMBER | STRING | paren.list.expression array.literal

| function.expression

paren.expression : ( assignment.expression )

paren.list.expression : paren.expression | ( list.expression , assignment.expression )

function.expression : function function.common | function identifier function.common

array.literal : [ ] | [ list.expression ]

postfix.expression : attribute.expression | full.postfix.expression

attribute.expression : simple.qualified.identifier | attribute.expression property.operator | attribute.expression

arguments

full.postfix.expression : primary.expression | full.postfix.expression property.operator | postfix.expression

++ | postfix.expression --

29

property.operator : brackets

brackets : { } | { list.expression }arguments : ( ) | paren.list.expression

unary.expression : postfix.expression | void unary.expression | ++ postfix.expression | -- post-

fix.expression | + unary.expression | - unary.expression | ~ unary.expression | ! unary.expression

multiplicative.expression : unary.expression | multiplicative.expression * unary.expression | multi-

plicative.expression / unary.expression | multiplicative.expression % unary.expression

additive.expression : multiplicative.expression | additive.expression + multiplicative.expression |additive.expression - multiplicative.expression

shift.expression : additive.expression | shift.expression << additive.expression | shift.expression >>

additive.expression | shift.expression >>> additive.expression

relational.expression : shift.expression | relational.expression < shift.expression | relational.expression

> shift.expression | relational.expression <= shift.expression | relational.expression >= shift.expression

equality.expression : relational.expression | equality.expression == relational.expression | equal-

ity.expression != relational.expression | equality.expression === relational.expression | equality.expression

!== relational.expression

bitwise.and.expression : equality.expression | bitwise.and.expression & equality.expression

bitwise.xor.expression : bitwise.and.expression | bitwise.xor.expression ^ bitwise.and.expression

bitwise.or.expression : bitwise.xor.expression | bitwise.or.expression | bitwise.xor.expression

logical.and.expression : bitwise.or.expression | logical.and.expression && bitwise.or.expression

logical.xor.expression : logical.and.expression | logical.xor.expression ^^ logical.and.expression

logical.or.expression : logical.xor.expression | logical.or.expression || logical.xor.expression

conditional.expression : logical.or.expression | logical.or.expression ? assignment.expression : as-

signment.expression

non.assignment.expression : logical.or.expression | logical.or.expression ? non.assignment.expression

: non.assignment.expression

assignment.expression : conditional.expression | postfix.expression = assignment.expression | post-

fix.expression compound.assignment assignment.expression | postfix.expression logical.assignment as-

signment.expression

compound.assignment : *= | /= | %= | += | -= | <<= | >>= | >>>= | &= | ^= | |=logical.assignment : &&= | ^^= | ||=list.expression : assignment.expression | list.expression , assignment.expression

statement.abbrev : expression.statement semicolon.abbrev | block | if.statement.abbrev | do.statement

semicolon.abbrev | while.statement.abbrev | for.statement.abbrev | continue.statement semicolon.abbrev

| break.statement semicolon.abbrev | return.statement semicolon.abbrev

statement.no.shortlf : expression.statement semicolon.no.shortlf | block | if.statement.no.shortlf |do.statement semicolon.no.shortlf | while.statement.no.shortlf | for.statement.no.shortlf | continue.statement

semicolon.no.shortlf | break.statement semicolon.no.shortlf | return.statement semicolon.no.shortlf

30

statement.full : expression.statement semicolon.full | block | if.statement.full | do.statement semi-

colon.full | while.statement.full | for.statement.full | continue.statement semicolon.full | break.statement

semicolon.full | return.statement semicolon.full

substatement.abbrev : empty.statement | statement.abbrev

substatement.no.shortlf : empty.statement | statement.no.shortlf

substatement.full : empty.statement | statement.full

semicolon.abbrev : ; | /* empty */

semicolon.no.shortlf : ; | /* empty */

semicolon.full : ;

empty.statement : ;

expression.statement : list.expression.allow.in

block : { directives }if.statement.abbrev : if paren.list.expression substatement.abbrev | if paren.list.expression sub-

statement.no.shortlf else substatement.abbrev

if.statement.no.shortlf : if paren.list.expression substatement.no.shortlf else substatement.no.shortlf

if.statement.full : if paren.list.expression substatement.full | if paren.list.expression substatement.no.shortlf

else substatement.full

do.statement : do substatement.abbrev while paren.list.expression

while.statement.abbrev : while paren.list.expression substatement.abbrev

while.statement.no.shortlf : while paren.list.expression substatement.no.shortlf

while.statement.full : while paren.list.expression substatement.full

for.statement.abbrev : for ( for.initializer ; optional.expression ; optional.expression ) substate-

ment.abbrev

for.statement.no.shortlf : for ( for.initializer ; optional.expression ; optional.expression ) substate-

ment.no.shortlf

for.statement.full : for ( for.initializer ; optional.expression ; optional.expression ) substate-

ment.full

for.initializer : /* empty */ | list.expression

optional.expression : list.expression | /* empty */

continue.statement : continue

break.statement : break

return.statement : return | return list.expression

directive.abbrev : empty.statement | statement.abbrev

directive.full : empty.statement | statement.full

directives : /* empty */ | directives.prefix directive.abbrev

directives.prefix : /* empty */ | directives.prefix directive.full

typed.identifier : identifier | identifier COLON type.expression

function.common : LP parameters RP result block

parameters : /* empty */ | nonempty.parameters

31

nonempty.parameters : parameter.init | nonempty.parameters , parameter.init

parameter : parameter.attributes typed.identifier

parameter.attributes : /* empty */ | constparameter.init : parameter

program : directives

32

References

[1] Good, M and Actor, G. ”Using MusicXML for File Interchange”. Proceedings of the Third Inter-

national Conference WEB Delivering of Music, IEEE Press, Los Almitos, CA, 2003, p. 153.

[2] Good, M, ”MusicXML for Notation and Analysis”. The Virtual Score: Representation, Retrieval,

Restoration, Walter B. Hewlett and Eleanor Selfridge-Field, eds., MIT Press, Cambridge, MA,

2001, pp. 113-124.

[3] Castan, G., Good, M. and Roland, P. ”Extensible Markup Language (XML) for Music Appli-

cations: An Introduction”. The Virtual Score: Representation, Retrieval, Restoration, Walter B.

Hewlett and Eleanor Selfridge-Field, eds., MIT Press, Cambridge, MA, 2001, pp. 95-102.

[4] Good, M. ”MusicXML: An Internet-Friendly Format for Sheet Music”. XML 2001 Conference

Proceedings(Orlando, FL, December 9-14,2001)

[5] Good, M. ”MusicXML in Practice: Issues in Translation and Analysis”. MAX 2002: Musical

Application Using XML, pp. 47-54.

[6] Finale, Coda Music Technology, http://www.codamusic.com/.

[7] Cakewalk Pro Audio, Cakewalk, Inc., http://www.cakewalk.com/.

[8] Sibelius, The Sibelius Group, http://www.sibelius.com/.

[9] Macromedia Flash MX, Macromedia Inc., http://www.macromedia.com/.

[10] MIDI Manufacturers Association, The Complete MIDI 1.0 Detailed Specification v96.1 (second

edition), 2001.

[11] Waldemar Horwat, ”Java Script 2.0”, 2003, http://www.mozilla.org/js/language/js20/.

[12] Information technology – Computer graphics and image processing – The Virtual Reality Modeling

Language(VRML), ISO/IEC 14772-1:1997 and ISO/IEC 14772-2:2002.

[13] Maya, Alias System Corp., http://www.alias.co.jp

33

graduate thesis - User Interface Research Group - IGARASHI

Documents

Transcript of graduate thesis - User Interface Research Group - IGARASHI