Approaching an Audience Model of Listening Experience · 2011. 7. 25. · Raw Emotional Intensity...

Approaching an Audience Model of Listening Experience

Finn Upham and Stephen McAdamsCIRMMT, Schulich School of Music, McGill University

Outline

• The Question: Is there stimulus related information in continuous response data?

• An answer: Measuring simultaneity of active responses across the audience:

• Testing coordination of audience activity: Goodness-of-Fit Test

• Testing coordination of alternating activities

• Testing coordination between two audiences’ activity

• Conclusions

Introducing the Data

• Ratings of Experienced Emotional Intensity

• Two audiences: 1st at live performance, 2nd watching audiovisual recording.

• Stimuli: Boston Symphony Orchestra performing orchestral works by Mozart.

• Overture to the Marriage of Figaro, K492

• Rondo from First Symphony, K16

• Adagio from Clarinet Concerto, K622

• Finale from Jupiter Symphony, K551

Is there information to be found in this data?

Continuous Ratings of Emotional Intensity from Two Audiences

0 50 100 150 2000

0.2

0.4

0.6

0.8

1

Emot

iona

l Int

esity

Time(s)

Raw Emotional Intensity Ratings of Audience 1 to the Marriage of Figaro Overture, K492

0 50 100 150 2000

0.2

0.4

0.6

0.8

1

Emot

iona

l Int

esity

Time(s)


Common summary of audience response

Q: How representative is this time series of individuals’ responses?

0 50 100 150 2000

0.2

0.4

0.6

0.8

1

Emot

iona

l Int

esity

Time(s)


0 50 100 150 2000

0.2

0.4

0.6

0.8

1Average Emotional Intensity Rating of Audience 1 to the Marriage of Figaro Overture, K492

Emot

iona

l Int

esity

Time(s)

Distinct Contours for the Same Stimulus

The average is not the whole story. Is there coordination?

0 50 100 150 2000

0.5


0 50 100 150 2000

0.51

Individual Emotional Intensity Ratings

0 50 100 150 2000

0.51

0 50 100 150 2000

0.51

0 50 100 150 2000

0.51

Time(s)

Activity Analysis

Measuring coordination of events across synchronized time series.

Counting the number of participants showing over 3% change in rating value over the same 2-second time window.

18 20 22 24 26 28 30 32 34 36 380

0.2

0.4

0.6

0.8

1

Emot

iona

l Int

esity

Time(s)


18 20 22 24 26 28 30 32 34 36 380

2

4

6

8

Increasing Activity in Audience 1 to the Marriage of Figaro Overture, K492

Time(s)

# pa

rticip

ants

incr

easin

g18 20 22 24 26 28 30 32 34 36 38

0

2

4

6

8Decreasing Activity in Audience 1 to the Marriage of Figaro Overture, K492

Time(s)#

parti

cipan

ts d

ecre

asin

g

18 20 22 24 26 28 30 32 34 36 38

5

0

5

Increasing and Decreasing Activity in Audience 1 to the Marriage of Figaro Overture, K492

Time(s)

Activ

ity L

evel

s

IncreasesDecreases

Audience Activity as an Alternative to the Average

Summarizing the Audience Response

0 50 100 150 2000

0.2

0.4

0.6

0.8

1

Emot

iona

l Int

esity

Time(s)


0 50 100 150 2000

0.2

0.4

0.6

0.8


Emot

iona

l Int

esity

Time(s)

0 50 100 150 20010

5

0

5

10

15Increasing and Decreasing Activity in Audience 1 to the Marriage of Figaro Overture, K492

Time(s)

# Ac

tive

Parti

cipan

ts

IncreasesDecreases

18 20 22 24 26 28 30 32 34 36 380

0.2

0.4

0.6

0.8

1

Emot

iona

l Int

esity

Time(s)

Emotional Intensity Ratings of Audience 1, Excerpt

18 20 22 24 26 28 30 32 34 36 380

2

4

6

8

Increasing Activity in Audience 1, Excerpt

Time(s)

# pa

rtici

pant

s in

crea

sing

0 1 2 3 4 5 6 7 8 9 100

2

4

6

8Increasing Activity Distribution of Audience 1, Excerpt

# tim

e in

terv

als

Activity Level

Coordination and Activity Levels

• Levels of Activity: Ratio of participants changing ratings in the same time interval.

• Rarely are most people responding at the same time, some people responding nearly all the time.

• Activity Distribution: Count the number of time intervals of each level of activity to evaluate the actual activity distribution for this audience and piece.

Coordination in Activity Distributions

• Ideally, all participants would respond together, or nearly.

• Random model: all participants respond independently, with equal likelihood of expressing an event at any moment. (Poisson distribution)

• Practically, we expect a mix of the two, some random noise on top of some degree of coordination.

0 5 10 15 20 25 300

10

20

30

Obviously Coordinated Activity Distribution

# tim

e in

terv

als

Activity Level

0 5 10 15 20 25 300

10

20

30

Completely Random Activity Distribution

# tim

e in

terv

als

Activity Level

0 50 100 150 2000

2

4

6

8

10

12Increasing Activity in Audience 1 to the Marriage of Figaro Overture, K492

Time(s)

Activ

ity L

evel

0 5 10 15 20 25 300

10

20

30

Increasing Activity Distribution for Audience 1 to K492

# tim

e in

terv

als

Activity Level

0 50 100 150 2000

2

4

6

8

10

12Increasing Activity in Audience 1 to Figaro Overture, K492

Time(s)

Activ

ity L

evel

0 5 10 15 20 25 300

10

20

30

Actual and Random Activity Distribution for Audience 1 to K492

# tim

e in

terv

als

Activity Level

ActualRandom

1 2 3 40

10

20

30

40

50Bins for Goodness of Fit Test for Increasing Activity, Aud 1, K492

# sa

mpl

es

Bins

2 =15.4903p < 0.002

ActualRandom

Testing the Distribution: Goodness of Fit Test

• Evaluate Coordination: Goodness-of-Fit Test of actual distribution against random model of all participants responding independently.

• Collapse distributions into small number of bins (4) for χ2 calculation.

• Assess significance with #bins - 1 degrees of freedom.

Testing Increasing Activity Distribution 1

• Example: distribution of increasing activity during K492 in Audience 1.

• 3 degrees of freedom • χ2 = 15.5, p < 0.002**• Null Hypothesis rejected!

• For this Audience, the pattern of increases in ratings of emotional intensity is coordinated enough to justify looking for causes (in the stimulus).

0 50 100 150 2000

2

4

6

8

10

12Increasing Activity in Audience 1 to Figaro Overture, K492

Time(s)

Activ

ity L

evel

0 5 10 15 20 25 300

5

10

15

20

25

30

35

Actual and Random Activity Distribution for Audience 1 to K492

# tim

e in

terv

als

Activity Level

ActualRandom

1 2 3 40

10

20

30

40


# sa

mpl

es

Bins

2 =15.4903

p < 0.002

ActualRandom

Testing Increasing Activity Distribution 2


• 3 degrees of freedom • χ2 = 5.4, p < 0.5• Null Hypothesis NOT rejected!

• For this piece, this audience did not show more coordination than would be expected from a random process. There is not sufficient reason to try to relate the response pattern to the music.

0 50 100 150 200 250 300 350 4000

5

10

15

Increasing Activity in Audience 2 to Adagio, K622

Time(s)

Activ

ity L

evel

0 5 10 15 200

10

20

30

40

50

60

Actual and Random Activity Distribution for Audience 2 to Adagio, K622

# tim

e in

terv

als

Activity Level

ActualRandom

1 2 3 40

20

40

60

80


# sa

mpl

es

Bins

2 =5.3664

p < 0.2ActualRandom

Alternating activity Coordination Test

• Two actions possible: Increasing and Decreasing of emotional intensity:

• Test to see if the audience is alternating between the two using a joint distribution.

• Null Hypothesis: Increasing activity and decreasing activity are independent of each other.

50 100 150 200 250 300 35010

5

0

5

10

15Increasing, Decreasing Activity in Audience 2 to Figaro Overture, K551

Time(s)

Activ

ity L

evel

IncreasesDecreases

0 5 10 15 200

10

20

30

40

50

Increasing Distribution for Aud 2 to K551

# tim

e in

terv

als

Activity Level0 5 10 15 20

0

10

20

30

40

50

Decreasing Distribution for Aud 2 to K551

# tim

e in

terv

als

Activity Level

0 5 10 15 200

10

20

30

40

50

Decreasing Distribution for Aud 2 to K551

# tim

e in

terv

als

Activity Level

0 10 20 30 40 500

5

10

15

20

Increasing Distribution for Aud 2 to K551

# time intervals

Activ

ity L

evel

5 10 15 20

5

10

15

20

Decreasing Activity

Incr

easi

ng A

ctiv

ity

2

4

6

8

10

12

14

1 2 3 40

10

20

30

40

50Goodness of Fit Bins for Alternating Activity, Aud 2, K622

# sa

mpl

e

Bins


• Null Hypothesis: Increasing activity and decreasing activity are independent of each other.

• Model distribution: if the activity were independent

• Evaluate differences between model and actual joint distributions of activity through four bins:• When mostly increasing• When mostly decreasing• When hardly active• Everything else


• Random model distribution: Increasing activity and decreasing activity are independent of each other.

• Ideal distribution: Participants rarely actively disagree on emotional force rating changes in any 2 second window.

• Four bins group joint activity levels to emphasize the difference between these two possibilities.

Col 1Matrix ideal, collapsed matrix ideal, bins Idealcol2Matrix Rand, collapsed matrix Rand, bins Rand

5 10 15 20

5

10

15

20

Example Random Joint Distribution

Increasing Activity Level

Dec

reas

ing

Activ

ity L

evel

0

2

4

6

8

10

1 2 3 40

10

20

30

40

50

60Bins for Goodness of Fit Test

# sa

mpl

e

Bins

5 10 15 20

5

10

15

20


Example Coordinated Alternating Joint Distribution

Dec

reas

ing

Activ

ity L

evel

0

2

4

6

8

10

1 2 3 40

10

20

30

40

50


# sa

mpl

e

Bins

Testing Alternating Activity Distribution 1

• Example: distribution of alternating activity during K551 in Audience 2.

• 3 degrees of freedom • χ2 = 15.1, p < 0.002• Null Hypothesis rejected!

• For this piece and audience, the patterns of increases and decreases in ratings appear to be related (i.e., alternating), supporting the assumption that the stimulus is driving both.

50 100 150 200 250 300 35010

5

0

5

10

15Increasing and Decreasing Activity in Audience 2 to K551 Finale

Time(s)

Num

ber o

f act

ive

parti

cipa

nts

IncreasesDecreases

5 10 15 20

5

10

15

20


Random Model Distribution

Dec

reas

ing

Activ

ity L

evel

0

2

4

6

8

10

5 10 15 20

5

10

15

20

Actual Joint Alternating Distribution


Dec

reas

ing

Activ

ity L

evel

0

2

4

6

8

10

1 2 3 40

10

20

30

40

50

60

70Bins for Goodness of Fit Test for Alternating Activity, Aud 2, K551

# sa

mpl

es

Bins

2 =15.1253

p < 0.002

ActualModel

Testing Alternating Activity Distribution 2


• 3 degrees of freedom • χ2 = 3.9, p < 0.5• Null Hypothesis NOT rejected!

• For this piece and audience, the patterns of increases and decreases in ratings appear to be independent of each other, suggesting that the stimulus did not succeed in driving a single simple emotional path.

50 100 150 200 250 300 350 40010

5

0

5

10

15Increasing and Decreasing Activity in Audience 1 to K622, Adagio

Time(s)

Num

ber o

f act

ive

parti

cipa

nts

IncreasesDecreases

10 20 30

5

10

15

20

25

30



Dec

reas

ing

Activ

ity L

evel

0

2

4

6

8

10

10 20 30

5

10

15

20

25

30

Actual Joint Alternating Distribution


Dec

reas

ing

Activ

ity L

evel

0

2

4

6

8

10

1 2 3 40

10

20

30

40

50

60

70Bins for Goodness of Fit Test for Alternating Activity, Aud 1, K622

# sa

mpl

es

Bins

2 =3.8751

p < 0.5

ActualModel

Two Audience Activity Coordination Test

• Two audiences responding to the same stimulus: Do they show the same pattern of activity over time?

• Like the Alternating Coordination Test, we can evaluate the joint distribution of the two activities.

0 50 100 150 2000

5

10

15Increasing Activity in Audience 1 and 2 to K492, Overture

Time(s)

Activ

ity L

evel

Audience 1Audience 2

0 10 20 300

5

10

15

20

25

30

35

Increasing Distribution for Audience 1 to K492

# tim

e in

terv

als

Activity Level

0 10 20 300

5

10

15

20

Increasing Distribution for Audience 2 to K492

# time intervals

Activ

ity L

evel

10 20 30

5

10

15

20


Aud 1 Activity Level

Aud

2 Ac

tivity

Lev

el

0

2

4

6

8

10


• Random model: Increasing ratings in one group have no relationship to the increasing ratings in the other.

• Ideal distribution: The level of activity of one audience matches the level of activity of the other for all 2 second windows.

• Four bins group joint activity levels to emphasize the difference between these two possibilities.

Col 1Matrix ideal, collapsed matrix ideal, bins Ideal

col2Matrix Rand, collapsed matrix Rand, bins Rand

10 20 30

5

10

15

20

Independent Joint Distribution


Aud

2 Ac

tivity

Lev

el

0

2

4

6

8

10

1 2 3 40

5

10

15

20

25

30


# sa

mpl

es

Bins

10 20 30

5

10

15

20


Coordinated Joint Distribution

Aud

2 Ac

tivity

Lev

el

0

2

4

6

8

10

1 2 3 40

10

20

30

40

50

60


# sa

mpl

es

Bins

Two Audience Activity Distribution 1

• Example: joint distribution of audiences’ increasing activity during K492.

• 3 degrees of freedom • χ2 = 20.8, p < 0.001• Null Hypothesis rejected!

• For this piece, the two audiences shared many moments of similar increasing activity, strengthening the claim that the ratings were driven by the stimulus.

0 50 100 150 2000

5

10

15Increasing Activity in Audience 1 and 2 to K492, Overture

Time(s)

# ac

tive

parti

cipa

nts

Audience 1Audience 2

10 20 30

5

10

15

20



Aud

2 Ac

tivity

Lev

el

0

2

4

6

8

10

10 20 30

5

10

15

20

Actual Joint Distribution


Aud

2 Ac

tivity

Lev

el

0

2

4

6

8

10

1 2 3 40

10

20

30

40

50Goodness of Fit Bins for Two Audience Increasing Activity, K492

# sa

mpl

es

Bins

2 =20.7624

p < 0.001

ActualModel

Results for this data set: Simple Activity

Table of results for increasing activity and decreasing activity. Against the random model of all participants having equal likelihood of changing ratings at any moment.

• Implications: • Increases in ratings are more

often significantly coordinated than decreases in ratings.

• Larger audience shows more coordination.

ActivityActivity Audience 1Audience 1 Audience 2Audience 2

Piece

K492

K16

K622

K551

Inc Dec Inc Dec

p< 0.002** 0.002** 0.001** 0.1

p< 0.025** 0.5 0.5 0.97

p< 0.05* 0.5 0.2 0.9

p< 0.025* 0.001** 0.05* 0.5

* satisfies α<0.05 **satisfies α<0.005

Results for this data set: Alternating Activity

Table of results for alternating increasing and decreasing activity against the random model of increasing activity being independent of decreasing activity and vice versa.

• Implications: • Some stimuli do not induce

alternating increases and decreases in ratings.

AlternatingAlternating Audience 1 Audience 2

Piece

K492

K16

K622

K551

Alt Inc/Dec Alt Inc/Dec

p< 0.001** 0.001**

p< 0.5 0.3

p< 0.5 0.15

p< 0.001** 0.002**


Results for this data set: Two Audience Activity

Table of results for coordination of increasing or decreasing activity between the two audiences, against a random model of being independent.

• Implications: • Increasing activity seem to

be more replicable than decreasing activity.

Two AudienceTwo Audience Increasing Decreasing

Piece

K492

K16

K622

K551

Aud1/Aud2 Aud1/Aud2

p< 0.001** 0.001**

p< 0.5 0.75

p< 0.02* 0.5

p< 0.02* 0.75


Conclusions

• Continuous rating data is not always sufficiently coordinated to justify interpreting its temporal variation with respect to the stimulus.

• If participants do not respond in parallel, audiences may or may not be coordinated on a larger scale.

• Different stimuli generate different degrees of coordination in the emotional intensity profiles of audience members-- not all stimuli are created equal!

• Increases in emotional intensity ratings seem to be more coordinated than decreases. This asymmetry should be considered further.

• Considering response patterns across an audience makes it possible to answer the question of coordination and ask many more.

My Thanks

• To Stephen McAdams for letting me play with this data for so long.

• To my current and former lab mates.

• To the various funding bodies that made the collection and analysis of this data possible: NSERC, SSHERC, and CRC.

• To SEMPRE and McGill GREAT travel award for allowing me to be present.

• To you all for sitting through 66 graphs.

Approaching an Audience Model of Listening Experience · 2011. 7. 25. · Raw Emotional Intensity...

Documents

Transcript of Approaching an Audience Model of Listening Experience · 2011. 7. 25. · Raw Emotional Intensity...