SE in RT Audio ApplicationsSE in RT Audio Applications

Bert Schiettecatte [bschiett@vub.ac.be]

Promotor: Prof. D. Vermeir

Co-promotor: Prof. R. Lauwereins

Advisors: S. Himpe & T. Vander Aa

IntroductionIntroduction

Sound: infinite discrete sequence of normalized numbers = ‘samples’

RT Audio Synthesis: real-time (RT) computation and playback of synthetic sound

Synthetic Sound: sound generated using a mathematical model

RT Deadlines: buffer scheduling, event handling

Goals & LimitationsGoals & Limitations

Not: – Detailed treatment of digital signal processing

(DSP) arithmetic for audio processing– Experiments on the performance of OO in RT

applicationsInstead:

– Realistic exposure to application of some SE techniques in a RT application

– Roadmap for building a RT audio synthesizer

Case StudyCase Study

Real-time audio synthesizer on handheld device: StrongARM running PocketPC

To be used by musicians on the roadGoals:

– Feasibility– Exposure to realistic RT application– Exposure to DSP– SE principles realistic in RT applications

ImplementationImplementation

Trade-offs:– Floating-point VS fixed-point– Double buffering, tripple buffering or

ring buffering of sound– Static VS dynamic networks– OO VS best-fit implementation

ImplementationImplementation

Strategy: bench & correctness of1. Arithmetic

2. Synthesis Algorithm

3. Buffer Scheduling & OS Timing

4. Event Scheduling

Fixed VS float arithmeticFixed VS float arithmetic

Arithmetic: major impact on overall performance & sound quality (quantization noise)

Benchmark for comparing floating-point / fixed-point performance: LINPACK

Case study HW: factor 10 speedup (assembly-level analysis reveals float arithmetic implementation: library)

Fixed-point arithmetic and GPFixed-point arithmetic and GP

Floating-point arithmetic: sometimes regarded as ‘unlimited range’ by developers

Fixed-point arithmetic: limited per definition, requires detailed range analysis in some algorithms, resulting in confusing code

Solution: use generative programming to derive best precision for a variable

Generalized: explicit type for a variable can be substituted for a set of constraints

Synthesis AlgorithmSynthesis Algorithm

Additive synthesis: sum (harmonically related) weighted sine waves

Envelope: piecewise linear function changing weight over time

Requirements: – Oscillator: efficient sine

algorithm (e.g. CORDIC)– Envelopes: linear

interpolation

x

+

x

x

.............

amplitude

amplitude

amplitude

f

2*f

n*f

Synthesis AlgorithmSynthesis Algorithm

Use fixed-point arithmetic to implement algorithm

Dynamic network: runtime sequencing of DSP blocks, buffer granularity

Static network: compile time sequencing of DSP blocks, sample granularity

Here: static network because of fixed-point arithmetic & inlining

Buffer Scheduling: OS APIBuffer Scheduling: OS API

In general: 1. Check device capabilities

2. Open device

3. Prepare buffer

4. Queue buffer in the OS for playback

5. Wait until playback done

6. Goto 3

7. Unprepare buffer

8. Close device

Buffer SchedulingBuffer Scheduling

Algorithms:– Naive– Double/Triple buffering– Ring buffering

Buffer Scheduling: NaiveBuffer Scheduling: Naive

Algorithm:1. Prepare buffer

2. Queue buffer in the OS for playback

3. Wait until OS signals playback done

4. Goto 1

Problems: – sound pauses (prepare + queue)– OS event is overhead

Buffer Scheduling: Double/TrippleBuffer Scheduling: Double/Tripple

Algorithm:1. Prepare buffer 12. Queue buffer 1 in the OS for

playback3. Wait until OS signals buffer 2 done4. Prepare buffer 25. Queue buffer 26. Wait until OS signals buffer 1 done7. Goto 1

Problems:– OS event overhead

W

S

Buffer Scheduling: Ring BufferBuffer Scheduling: Ring Buffer

Algorithm:1. Fill ½ & queue buffer for looping

playback

2. Get playing position of the OS in buffer

3. Fill buffer from the last known rendering position up to the OS position

4. Update rendering position

5. Wait ¼ buffer size

6. Goto 2

Problems:– Tricky: sleep accuracy (WinCE 3.0: 1 to

2 ms VS Win98: 20ms!)– Choosing wait time is hard

W S

Buffer Scheduling: RealityBuffer Scheduling: Reality

Scheduling: very hard– Requires inspiration and experimenting

(e.g. buffer size problems in WinCE 3.0)– Buffer underrun detection/avoidance

(e.g. MP3)

Event SchedulingEvent Scheduling

Possibilities: – Seperate event thread updating settings– Handle events in the synthesizer class: keep a song

pointer & update when necessary

Seperate thread: requires additional CPU time for scheduling, synchronization

In the synth class: better, keep a song pointer (in samples) and update settings based on the song’s tempo (e.g. 120 BPM = update every 0.5s * 22050 Hz = 11025 samples)

Event SchedulingEvent Scheduling

Latency: the delay on events caused by the buffer scheduler

Latency:– In seperate event thread: because

of synchronization, at least 1 sound buffer

– In synthesis loop: almost none, since events can be handled almost every sample

Implementation: Best-fitImplementation: Best-fit

In general: mix paradigms to write clear code

OO not always the best choice: depends on levels of re-use (e.g. OS timer)

Machine-level inspection of code: crucial when using OO in RT applications

OO not per definition a bad thing

Implementation: Object-orientednessImplementation: Object-orientedness

Here: OO is the right paradigm when– It doesn’t keep the compiler from optimizing – It entirely disappears after compilation (e.g.

inlining): no run-time OO– It simplifies code

In general: OO languages without real ‘compile-time’ tend to have performance problems

Meta-programming: welcome addition

ContributionContribution

This dissertation: one of the few detailed available roadmaps for RT audio synthesis implementation

Not much new here (but still large amount of research, everything proprietary)

Application of generative programming to audio processing and fixed-point arithmetic

ConclusionConclusion

Implementing a RT audio synthesizer is hard Series of benchmarks & tests crucial OO design has to be applied with common sense

and is not always the best choice Generative programming can be a powerful

mechanism to abstract at little or no cost A meta-programming level adds expressive power Compiler features (inlining, templates, …)

essential

Synthesis engine demonstrationSynthesis engine demonstration

Sets up the additive synthesis engine, starts the sound buffer scheduler, loads events from a text file and feeds them to the additive synthesizer

81600 0 1 130.8120 0 0 11 0 1 184.9981 0 0 12 0 1 220.00...

Time Component ID

Parameter ID ParameterValue