D’ IMATGE I GRÀFICS!T

EC

NO

LO

GIE

S!

D’ IMATGE I GRÀFICS!

TE

CN

OL

OG

IES!

http://gilab.udg.edu

E-cecilia: implementation of a music game

R. García1,5, I. Barbancho2, L. Tardón2, J. Arambarri3, M. Magdics1,4, M. Sbert1

1University of Girona 2University of Malaga

3Virtualware 4Budapest University of Technology and Economics

5iMinds / University of Hasselt

D’ IMATGE I GRÀFICS!T

EC

NO

LO

GIE

S!

•  Introduction •  Game description

•  Architecture, client-server model

•  Server

•  Music composition & analysis

•  Client •  Graphics, non-photorealistic rendering

•  Natural interaction

•  Immersive devices

Contents  

D’ IMATGE I GRÀFICS!T

EC

NO

LO

GIE

S!

Introduc-on  

•  Learning to sing requires long practising hours

•  The early stages use very simple songs → boring

•  Good singers start very young, so they tire more easily

•  Solution: Make the practising into a game

•  Related work:

•  SingStar: full songs, too complex for starting young children

•  Other games:

•  focused on keyboard / mouse interaction

•  don't do audio analysis

D’ IMATGE I GRÀFICS!T

EC

NO

LO

GIE

S!

•  Serious game to practice singing and rhythm

•  The user is presented with a small village to explore

•  The houses are small, and can be enhanced by the user:

•  After clicking a house, a song is presented

•  The user sings to the best of his abilities

•  The house evolves in response to the user singing

•  The user can continue exploration and singing until he has practiced enough.

Game  descrip-on  

D’ IMATGE I GRÀFICS!T

EC

NO

LO

GIE

S!

Gameplay  video  

D’ IMATGE I GRÀFICS!T

EC

NO

LO

GIE

S!

System  architecture  

•  Three main components: •  Game client (Unity)

•  Assets (on a web server)

•  Recommendation and Evaluation server

•  Audio and rhythm evaluation (matlab)

•  Recommendation system (C++ library)

•  Interconnection:

•  TCP sockets (game-server)

•  HTML (game-webserver)

D’ IMATGE I GRÀFICS!T

EC

NO

LO

GIE

S!

Game  client  

•  Object oriented, asynchronous architecture using the Unity game engine. Object behaviour uses C# scripts.

•  The main objects are:

•  World Creation (terrain, house creation and evolution)

•  World View: Camera, NPR effects, Dome transforms, GUI

•  World control: Kinect, mouse interaction

•  Asset downloading, communication with web server.

•  Microphone: song recording

•  ServerConnect: Communication with the server, evolving the GameState

D’ IMATGE I GRÀFICS!T

EC

NO

LO

GIE

S!

Music  server  

•  The game server is responsible for the automatic generation of musical excercises (songs) and the assessment of the user performance

Server

D’ IMATGE I GRÀFICS!T

EC

NO

LO

GIE

S!

Music  synthesis  

•  A set of training songs is used and analyzed

•  Created songs replicate the selected style and complexity of that of the corresponding exemplars

Training Generation

D’ IMATGE I GRÀFICS!T

EC

NO

LO

GIE

S!

Music  assessment  

•  Tempo, rythm and melody is compared against the reference

•  Works with inexpensive microphones too

D’ IMATGE I GRÀFICS!T

EC

NO

LO

GIE

S!

Graphics,  Non-­‐photorealis-c  Rendering  (NPR)  

•  Post processing stylization

–  State-of-the-art image processing (on the color and depth)

•  Flexible, a wide range of effects

•  Commercial Unity package under finalization

Original rendering (“Photo-realistic”)

Color Comic: Image simplification + edges

D’ IMATGE I GRÀFICS!T

EC

NO

LO

GIE

S!

NPR:  edge  effects  

Intensity + depth-normal edges Colored painting: edge color is driven by the (original) rendered color

D’ IMATGE I GRÀFICS!T

EC

NO

LO

GIE

S!

•  Replacement of the shadowed pixels by another color

NPR:  Shadow  effects  

Complementary shadows: Shadow color is the complementary color

of the shadowed surface

D’ IMATGE I GRÀFICS!T

EC

NO

LO

GIE

S!

•  Increasing the level of abstraction by depth

NPR:  depth  sensa-on  

Depth-based desaturation Depth-based simplification: Texture details are removed

in the background

D’ IMATGE I GRÀFICS!T

EC

NO

LO

GIE

S!

NPR  demo  

D’ IMATGE I GRÀFICS!T

EC

NO

LO

GIE

S!

•  Augmented reality: mobile device, augment camera image with controls (e.g. buttons)

•  Kinect: exergame (imitating real movements, like walking or turning) or control by hand movements

Natural  interac-on  

AR button Exergame: walk

D’ IMATGE I GRÀFICS!T

EC

NO

LO

GIE

S!

•  Different projection/camera settings

–  E.g. stereo: 2 cameras, dome: 360º projection with cube-map

Stereo, top-bottom Stereo, side-by-side

Immersive  visualiza-on  

D’ IMATGE I GRÀFICS!T

EC

NO

LO

GIE

S!

Immersive  visualiza-on  

Spherical dome

source: immersivedisplay.co.uk

Immersapod

D’ IMATGE I GRÀFICS!T

EC

NO

LO

GIE

S!

Immersive  visualiza-on  

•  NPR+immersive: NPR on the final image

–  Regular 2D image processing

•  No added computational complexity for using immersive displays w.r.t. NPR

Dome+color comic Immersive+B&W comic

D’ IMATGE I GRÀFICS!T

EC

NO

LO

GIE

S!

Conclusions  

•  Music teaching game •  State-of-the-art automatic music synthesis and

singing assessment

•  Controllable graphics style (NPR) as post-processing

•  Support for natural interaction, immersive visualization

D’ IMATGE I GRÀFICS!T

EC

NO

LO

GIE

S!

Thank you!

Questions?

D’ IMATGE I GRÀFICS!

TE

CN

OL

OG

IES!