Eng. 100: Music Signal Processing

DSP Lecture 8

Project 3: Music signal processing team project

Example: Music synthesizer / transcriber

Curiosity: http://web.engr.illinois.edu/~paris/demos

Announcements: Extension! Project 2 report.pdf due this week + one day (5pm) Read Project 3 description (Canvas) before lab this week Project 2 questions?

1

http://web.engr.illinois.edu/~paris/demoshttps://umich.instructure.com/

Outline

Part 0: Team project deliverables Project 3 classic

Part 1: Specifications for a baseline music synthesizer song.wav

Part 2: Specifications for a baseline music transcribersong.wav transcriber

0 610

1

2

3

4

Part 3: Transcriber approaches (new DSP methods)

8000 8050 8100 8150 8200 8250 8300 8350 8400 8450 85001

0.5

0

0.5

1

n (sample)

x[n

] =

x(n

/S)

Trumpet playing high G

pitch identification 787.5 Hz

This project is an engineering design project, so there is considerable room for creativity and originality.

(lifetime of learning...)2

F13 Projects

Spectrum modifications Effects processor: flange / reverb / pitch shift Effects processor: echo (delay) / flange / tone control Audio equalizer (bass boost etc.) Track mixer (with volume and tone controls) Pitch shifter (with effects: echo / chipmunk ...) Pitch and tempo shifter with phase vocoder Flute autotuner Spectrum analyzers Pitch analyzer and tuner Tuner and metronome Synthesizers Synthesizer with several instruments and tone controls Synthesizer with attack/decay controls and chords Synthesizer with selectable harmonics and other effects (tremolo, ET) Hammond B-3 organ synthesizer (with vibrato)Other Audio compressor Speech recognition system (spoken numbers)

3

F13 example GUI

4

F14 Projects

Spectrum analyzers / modifiers Pitch / tempo shifter Pitch tracker / synthesizer Real-time effects (flange, reverb, etc.)

Synthesizers Synth with controls Synthesizer / transcriber with rests and chords Synth with controls Synth with 8-bit sounds and random composer / transcriber Synth with keyboard input Loop synth with many options Polyphonic looping synth Synthesizer with chords / transcriber Drum machine / tempo tracker

5

F14 example GUI

6

Project 3 deliverables

P3 pitch in Thu. Nov. 5 class P3 proposal (40) P3 progress report / draft executive summary (60) P3 draft oral presentation (10) P3 final oral presentation (100)

Must include live demo of project

P3 peer evaluation (10) P3 final report and code (150)

See course schedule for deadlines.

This may be only a partial list; refer to course syllabus and further instructions.

7

Project 3 classicPart 1: Music synthesizer

8

Synthesizer overview

Synthesize (at least) one octave (G to G) Instrument choices (at least 5): Electric guitar Trumpet Clarinet Tone Brass section Your teams own sound(s), at least one of which must use additive synthesis

Synthesizer GUI Pull-down menu to select the instrument Note durations (at least 3): whole, half, quarter

Polyphony / mixing (optional) Create individual tracks separately for different instruments. Add them together in Matlab,e.g., x = 0.7 * x_guitar + 0.4 * x_trumpet. Use soundsc not sound to play the sum signal! The transcriber is not required to handle polyphony

(but that would be impressive and has been tackled by some past teams).9

Music synthesis: Instruments

Download project3.wav from web site (3.4 Mb).[x, S] = audioread('project3.wav'); soundsc(x,S)

Contents: Sound snippets: Length N = 32768 each, S = 44100 Hz. Electric guitar, clarinet, trumpet, tone; 13 notes each.

Additive synthesis: Create your own instrument. Be creative! Brass section: Reverb (add delayed copies) of trumpet.

10

Music Synthesizer: Instrument selection

Pull-down menu (aka pop up menu) to select instrument.h = uicontrol('style', 'popup', 'position', [1 1 100 50],...'string', 'guitar|clarinet|trumpet|tone|teamname|band');

To determine which instrument is selected (of 6 options here):v = get(h, 'value'); v==1 = guitar; v==2 = clarinet; v==3 = trumpet, etc. The variable h is called a handle to the GUI object.

Why is handle h needed? ??

The get function extracts the value for the particular GUI objectcorresponding to the handle h.

11

Music Synthesizer: Note durations

Use the following note durations.

Note Whole Half Quarter 1 secondLength 32668+100 16284+100 8092+100 S = 44100

Simplification: the final 100 samples of each note can be zeros,to provide duration information to transcriber.Is 100 zeros a problem musically? ??

Example of music transcriber output with note duration spacing:

1 5 7 9 15 17 21 41 43 57 61

0

1

2

3

4

time [quarter notes]

12

Music Synthesizer: GUI example

Simplification: performer selects duration manually.Clicking end button causes song to be playedand writes song to a .wav file. (Do not use a .mat file!)

13

Music Synthesizer: GUI example on Mac/Matlab

Suggestion: Start with just a few notes; add more later.Use for loops and anonymous functions (explained later).

14

Music Synthesizer: Reverb

To add reverb (reverberation, i.e., echo) to a sampled signal xwith N=length(x) try this:y = x(1:N-2*D) + x(1+D:N-D) + x(1+2*D:N);

Try delay D 1000. (units?) Use many more than 3 echoes to make it sound good. Random delay values can sound more realistic. Example. play

Trumpet signal x followed by reverberated signal y

Somewhat like a marching band or trumpet section with multi-ple trumpets.

Later lectures will discuss more advanced music synthesis techniques.15

Synthesizer summary

Minimum features: At least one octave Multiple instruments (at least 5) At least one instrument based on Fourier synthesis (explained later) At least 3 note durations: quarter, half, whole 100 samples of zeros after each note to aid transcriber is allowed

16

Part 2: Music Transcriber

17

Transcriber overview

Reads .wav file produced by your synthesizer.[x, S] = audioread('song.wav');

Generates musical staff notation as in Project 1. BUT: Also depicts note duration by separation. BUT: Must work for music sounds, not just sinusoids!

Otherwise, same as Project 1 transcriber specs. Does not have to include bass clef for guitar. Does not have to work for polyphonic music. ALSO: Error rate versus SNR plot, as in Project 2,

for at least 5 instruments.

18

Music Transcriber: Specifications

Output: Musical scale & notes using stem (Project 1) Duration: Shown in output by separation between notes:

Note Type Whole Half QuarterSeparation 3 extra spaces 1 extra space 0 extra space

Use reshape (more details in later lecture) columns ending in 100 zeros help indicate note lengths Find indexes I of those columns; can ignore the other columns for finding frequencies f . As in Project 1: frequencies = MIDI value =

table lookup to specify vertical staff position stem(I,table_lookup)

Do not need display the (bass) guitar tones. Do need an error rate versus SNR plot, as in Project 2.

19

Example error rate vs SNR plot

30 25 20 15 10 5 0 50

10

20

30

40

50

60

70

80

SNR [dB]

Err

or

Rate

[%

]

trumpet

tone

clarinet

TeamSound1

band

(These are hypothetical values for illustration only.) Put at least 5 instruments on one plot. (Extra instruments not required here.) Vary the SNR enough that the error rates range from 0% to at least 50% for every instrument. Is

the example plot above adequate? ??

20

Part 3: Transcriber approaches

21

Transcriber possibilities: Overview

Previous methods: Arccos ?? Spectrogram: Look for peaks.

Conceptually simple, but can be hard to program.

Correlation: using a set of candidate frequencies (P2)

New methods: Autocorrelation of segment with itself:

y[m] = n x[n]x[nm] has sharp peak at m=period. Harmonic product spectrum:

Down-sample spectra multiple ways and multiply;this process emphasizes 1st harmonic.

Combinations / variations of the above methods.

All of these methods have been used previously.

22

Spectrogram of The Victors: Sinusoids

"note index"

fre

qu

en

cy in

de

x (

k+

1)

N=16384 S=32768

Spectrogram of "The Victors:" Sinusoids

1 601

197

221

248

263

295

331

play

Durations are evident, except for two subsequent sinusoids of the same frequency.

23

Spectrogram of The Victors: Trumpet

"note index"

fre

qu

en

cy in

de

x (

k+

1)

N=8192, S=44100

Spectrogram of "The Victors:" Trumpet

1 601

82

600

play

Finding fundamental frequency of each note is possible but more challenging.

24

Example: Trumpet Waveform

8000 8050 8100 8150 8200 8250 8300 8350 8400 8450 85001

0.5

0

0.5

1

n (sample)

x[n

] =

x(n

/S)

Trumpet playing high G

Pitch detection: Find period of periodic waveform. Given S = 44.1kHz. By eye/hand: what is frequency here? ?? How can we automate? (Think about what we did by eye.)

25

Autocorrelation method

Recall: the correlation of one signal x[n] with another signal y[n] is thesum of their product: correlation = n x[n]y[n].

We define the autocorrelation of a x[n] to be the correlation of x[n] witha time-shifted version of itself, i.e., the other signal is a time-shiftedversion of x[n].

8000 8050 8100 8150 8200 8250 8300 8350 8400 8450 85001

0.5

0

0.5

1

n

x[n

] =

x(n

/S)

x[n]

x[n10]

The auto-correlation value depends on the shift amount m:autocorr[m] = n x[n]x[nm]. (Units of shift m? ??)

26

Autocorrelation illustrated

8000 8050 8100 8150 8200 8250 8300 8350 8400 8450 85001

0.5

0

0.5

1

n

x[n

] =

x(n

/S)

x[n]

x[n10]

8050 8100 8150 8200 8250 8300 8350 8400 8450 8500

0.5

0

0.5

n

x[n

] *

x[n

1

0]

sum = 21.61

Given x= [x[1] x[2] . . . x[N]], autocorr[10] = Nn=11 x[n]x[n10]In Matlab: sum( x(11:end) .* x(1:end-10) )

27

Autocorrelation illustrated

8000 8050 8100 8150 8200 8250 8300 8350 8400 8450 85001

0.5

0

0.5

1

n

x[n

] =

x(n

/S)

x[n]

x[n56]

8050 8100 8150 8200 8250 8300 8350 8400 8450 8500

0.5

0

0.5

n

x[n

] *

x[n

5

6]

sum = 50.06

autocorr[m] = n x[n]x[nm] is large when shift m = a period.

For which shift is autocorrelation largest? ??28

Autocorrelation properties

Maximum value of autocorrelation is n x2[n], i.e.,autocorr[m] =

nx[n]x[nm]

nx2[n].

For which m does autocorr[m] equal that largest value? ??

Fast way to compute autocorrelation for all shift values m:autocorr = real(ifft(abs(fft(x,2*length(x)).^2));

(Take EECS 351 to learn why these FFT commands yield autocorrelation values.)

29

Example: Trumpet autocorrelation

-1000

0

1000

2000

1 57 300

autocorrelation

shift (m+1)

autocorr = real(ifft(abs(fft(x,2*length(x)).^2)));

plot(autocorr(1:300), '.-')

To find frequency use f = 1/T where T = m = m/S so f = S/m.

Highest peak always at m = 0 (array index 1 in Matlab)Second largest peak away from zero at index = m+1 = 57 = T = m

S= Frequency = 1

period=

Sm=

44100 Hz571

= 787.5 Hz (high G)

30

Example: Trumpet spectrum

1 587 1168 30000

0.05

0.1

0.15

0.2

frequency index (k+1)

sp

ectr

um

By Project 1, the fundamental frequency is

f =kN

S =587132768

44100 Hz = 788.7 Hz (high G).

BUT: here the 2nd peak is higher than the 1st peak.= Using simple max function may locate the wrong peak!If we do not use max how do we avoid small noise peaks?

31

Harmonic product spectrum (HPS)

Multiply spectrum by down-sampled versions of itself.F = 2/N*abs(fft(x));

hps = F(1:5000) .* F(1:2:10000) .* F(1:3:15000); plot(hps)

1 587 1168 30000

0.5

1x 10

4

frequency index (k+1)

harm

onic

pow

er

spectr

um

HPS emphasizes harmonics over other stuff,so easier to find peaks in noise.

Now the highest peak matches the fundamental frequency. (HPS may not always work as well as in this example.) Note that we down-sampled the spectrum, not the signal !

32

Transcriber: Other helpful (?) ideas

Filtering out unwanted signals (noise) If we are interested in only 1 octave of pitches,

then can filter out all signals not in that octave (Lab 3). Can help for some approaches, but not for others.

Transcriber: frequencies

Prof. Yagle generated the sounds in project3.wavusing Circle of Fifths and multirate filtering (advanced topics).

Frequencies differ slightly from equal temperament tuning. If needed, use the tone signal to determine tuning. Alternatively, round MIDI number to the nearest integer:

MIDI = 69 + round (12 log2(frequency in Hertz/440))

33

Project 3 challenges

The octave problem:how to distinguish G (392 Hz) from G (784 Hz).(Trumpet has this problem.)

Use pattern recognition to identify instrument type from the patternof harmonics? Not required, but will impress...

Your team must sell/defend your choice of method(s)in both your final oral presentations and written reports.

34

Conclusion

I am not telling you how to do this project!Music transcription is not a solved problem;different approaches have pros/cons.

Apply what you have learned in the course. Research on music synthesis/transcription.

Pitch detection algorithms (e.g.):http://cnx.org/content/m11714/latest/

Your presentations of the results (tech comm)are as important as results themselves.This is very realistic for real-world engineering.

35

http://cnx.org/content/m11714/latest/

fd@rm@0: fd@rm@1: fd@rm@2: