Spatial Sound in X3D/X3DOM Athanasios G. Malamos

Multimedia LabDept. of Informatics Engineering

TEI of Crete, Heraklion, Crete, Greece(amalamos@ie.teicrete.gr)

18/06/2015

FUTURE

The initial question is, does X3D have to interface to the W3C API or just leave engineers to use it in the application level….. ??

(provocation!! )

A Hint……

The presence of virtual reality hardware interfaces (ex. VR Headsets) increases the impact of spatial sound…..

19TH INTERNATIONAL CONFERENCE ON 3D WEB TECHNOLOGYAUGUST 8-10 – VANCOUVER, CANADA

2014

X3D v3.4 and sound

In V3.4 could be included some neighbor to v3.4 features of W3C API

……………….

And perhaps in v.4 to reform the X3D sound components more drastically considering also OpenAL, FMOD, Creative's EAX, Microsoft's XACT Audio, etc….

19TH INTERNATIONAL CONFERENCE ON 3D WEB TECHNOLOGYAUGUST 8-10 – VANCOUVER, CANADA

2014

we decided to start with a test-bed where we could make all the compatibility tests

http://medialab.teicrete.gr/minipages/x3domAudio/index.html

Overview

Sound Nodes in X3D

Sound Nodes in X3DOM Framework

Test bed demo

Test bed methodology

20th International Conference on 3D Web Technology June 18-21 –Heraklion,Crete,Greece

Sound Nodes in X3D (1)

Abstract types: X3DSoundNode (the base for all sound nodes) X3DSoundSourceNode (this derive node types that can emit audio data)

Node reference: AudioClip is derived from the X3DSoundSourceNode. It specifies audio data

that can be referenced by the Sound node. Sound is derived from the X3DSoundNode. It is designed for the description

of the X3D scene sounds. Specifically, it determines both the location and the behavior of the sound.

20th International Conference on 3D Web Technology June 18-21 –Heraklion,Crete,Greece

Sound Nodes in X3D (2)

Inheritance diagram of X3D AudioClip Node

X3D: X3DChildNode

X3D: X3DNode

X3D: AudioClip

MFString url

SFTime duration_changed

SFString description

SFTime resumeTime

SFTime stopTimeSFTime startTime

SFFloat pitch SFTime pauseTimeSFNode metadata SFBool loop

SFTime elapsedTimeSFBool isActiveSFBool isPaused

X3D: X3DSoundSourceNodeSound

SFTime stopTime SFTime duration_changed

SFVec3f direction

SFFloat pitch

SFTime startTime SFTime resumeTime

SFNode metadataSFTime pauseTime

SFBool loop

SFTime elapsedTimeSFBool isActiveSFBool isPaused

X3D: X3DTimeDependentNode

X3D: X3DUrlObject

20th International Conference on 3D Web Technology June 18-21 –Heraklion,Crete,Greece

Sound Nodes in X3D (3)

X3D: Sound

SFFloat prioritySFNode source

SFVec3f direction

SFFloat intensity

SFFloat minFrontSFFloat minBack

SFNode metadataSFFloat maxFrontSFFloat maxBack

SFVec3f location

SFBool spatialize

X3D: X3DSoundNode

SFNode metadata

X3D: X3DChildNode

X3D: X3DNode

Inheritance diagram of X3D Sound Node

20th International Conference on 3D Web Technology June 18-21 –Heraklion,Crete,Greece

Sound Nodes in X3DOM (1)

X3DSoundNode, X3DSoundSourceNode, AudioClip and Sound nodes have been registered in X3DOM, but without the most of the properties of the respective X3D nodes.

Specifically, an X3DOM scene could include an audio file for playing, without distinguished spatial characteristics.

The attribute “SFBool enabled” has been extra added to X3DOM in comparison with X3D, which specifies whether the clip is enabled or not.

20th International Conference on 3D Web Technology June 18-21 –Heraklion,Crete,Greece

Sound Nodes in X3DOM (2)

X3D X3DOM X3D X3DOM

X3DSoundSourceNode: AudioClipAttributes

SFString description SFTime stopTime

SFBool loop SFTime duration_changed

SFNode metadata SFTime elapsedTime

SFTime pauseTime SFBool isActive

SFFloat pitch SFBool isPaused

SFTime resumeTime MFString url

SFTime startTime SFBool enabled

AudioClip Node in X3D – X3DOM

20th International Conference on 3D Web Technology June 18-21 –Heraklion,Crete,Greece

Sound Nodes in X3DOM (3)

X3D X3DOM X3D X3DOM

X3DSoundNode: SoundAttributes

SFVec3f location SFNode metadata

SFVec3f direction SFFloat intensity

SFFloat maxBack SFFloat priority

SFFloat maxFront SFNode source

SFFloat minBack SFBool spatialize

SFFloat minFront

Sound Node in X3D – X3DOM

20th International Conference on 3D Web Technology June 18-21 –Heraklion,Crete,Greece

Test Bed Methodology

We investigated the combination of X3D sound nodes and the structure of Web Audio API, in order to upgrade the X3DOM framework with audio spatial abilities for a 3D web scene.

X3D Audio Nodes

Custom Nodes

Web Audio API

Investigated Method

20th International Conference on 3D Web Technology June 18-21 –Heraklion,Crete,Greece

Why Web Audio API

It is open source and be supported from the most browsers. It allows the web developers to manage audio using JavaScript. It offers high-level sound abilities as filters, delay lines, amplifiers and spatial

effects (such as panning). Audio channels can have 3D distribution according to the position, speed or

direction of the viewer and the sound source.

20th International Conference on 3D Web Technology June 18-21 –Heraklion,Crete,Greece

Analysis of test bed methodology

The attributes of AudioClip (X3D node) and Sound (X3D node) have been re-introduced in X3DOM with the use of : Two enhanced custom nodes

• AudioSound• AudioSource

An Added Value set of Web Audio API nodes

20th International Conference on 3D Web Technology June 18-21 –Heraklion,Crete,Greece

Test Bed at a glance

Registered Web Audio API Nodes in X3DOM

AudioDestinationNode

AudioBufferSourceNode

GainNode

BiquadFilterNode

DelayNode

AudioListener

PannerNode

StereoPanner

ConvolverNode

AnalyserNode

ChannelSplitterNode

ChannelMergerNode

DynamicsCompressorNode

WaveShaperNode

OscillatorNode

Custom Nodes

AudioSound

AudioSource

AudioSource:X3DOM:X3DSoundSourceSFBool loopSFNode metadataX3D: X3DSoundSourceNodeSFString descriptionMFString urlSFTime pauseTimeSFFloat pitchSFTime resumeTimeSFTime startTimeSFTime stopTime

Test b

ed

20th International Conference on 3D Web Technology June 18-21 –Heraklion,Crete,Greece

X3D to test bed matching

20th International Conference on 3D Web Technology June 18-21 –Heraklion,Crete,Greece

NodeX3D X3DOM Test bed

Attribute Web Audio API Custom Node

X3DSoundSourceNode: AudioClip

SFString description AudioSourceSFBool loop AudioBufferSourceNode AudioSourceSFNode metadata AudioSource

SFTime pauseTimeAudioBufferAudioContext

AudioSource

SFFloat pitch AudioBufferSourceNode AudioSource

SFTime resumeTime AudioBufferAudioContext

AudioSource

SFTime startTime AudioBufferSourceNode AudioSourceSFTime stopTime AudioBufferSourceNode AudioSourceSFTime duration_changedSFTime elapsedTimeSFBool isActiveSFBool isPausedMFString url AudioSource

SFBool enabled

X3D to Test bed matching (2)

20th International Conference on 3D Web Technology June 18-21 –Heraklion,Crete,Greece

Node

X3D X3DOM Investigated Method

Attribute Web Audio API Custom Node

X3DSoundNode:

Sound

SFVec3f location PannerNode PannerNode (New Registered Node)

SFVec3f direction PannerNode PannerNode (New Registered Node)

SFFloat maxBack PannerNode PannerNode (New Registered Node)

SFFloat maxFront PannerNode PannerNode (New Registered Node)

SFFloat minBack PannerNode PannerNode (New Registered Node)

SFFloat minFront PannerNode PannerNode (New Registered Node)

SFNode metadata

SFFloat intensity GainNode

SFFloat priority DynamicsCompressorNode

SFNode source AudioSource

SFBool spatialize TRUE TRUE

All layers overview of the matching (1)

20th International Conference on 3D Web Technology June 18-21 –Heraklion,Crete,Greece

Web Audio API

AudioBufferSourceNode

loopplaybackRate

Custom Node

AudioSource

sourcedescription

Web Audio API

AudioContext

currentTime

Web Audio API

AudioBuffer

length

loopmetadataurlpitchpauseTimeresumeTimestartTimestopTime

X3DSoundSourceNode

X3D: AudioClip

SFString descriptionSFBool loop

SFTime resumeTime

SFNode metadata

SFFloat pitch SFTime pauseTime

SFTime startTime

SFTime duration_changedMFString url SFTime stopTime

SFTime elapsedTimeSFBool isActiveSFBool isPaused

start()stop()

destinationlistenersampleRatemozAudioChannelType

sampleRatedurationnumberOfChannels

Custom AudioSound Node

All layers overview of the matching (2)

20th International Conference on 3D Web Technology June 18-21 –Heraklion,Crete,Greece

X3DSoundNode

X3D: Sound

SFVec3f location SFVec3f direction

SFFloat intensity

SFFloat maxBack

SFNode metadataSFFloat maxFront

SFFloat minBack

SFNode source SFFloat priority SFFloat minFront

SFBool spatialize (default true)

X3DOM Registered Node

PannerNode

positionorientation

Web Audio API

DynamicsCompressorNode

priority

Web Audio API

GainNode

gain

Custom Node

AudioSource

source

coneInnerAngleconeOuterAngleconeOuterGaindistanceModelmaxDistancepanningModelrefDistancerolloffFactor

threshold knee ratioreductionattack release

velocity

Web Audio API

PannerNode

position()orientation()velocity()

Custom AudioSound Node

Future work

•To test more technologies and get the best performance

•To cooperate more with members of the community that are interested to extend and enhance X3D audio api.

End Of PresentationTHANK YOU!!!