Thai Intonation in Four Emotions Apiluck Tumtavitikul and Kanlayarat Thitikannara Linguistics Dept.,...

Thai Intonation in Four Thai Intonation in Four EmotionsEmotions

Apiluck Tumtavitikul Apiluck Tumtavitikul andand Kanlayarat Kanlayarat ThitikannaraThitikannara

Linguistics Dept., Kasetsart UniversityLinguistics Dept., Kasetsart UniversityBangkok, Thailand 10900Bangkok, Thailand 10900email: email: [email protected]@ku.ac.th

The ICSTLL-39, Seattle, Washington, September 15-17, 2006

Tumtavitikul & Thitikarnnara 2006Tumtavitikul & Thitikarnnara 2006

IntroductioIntroductionn

Emotions are universally Emotions are universally expressed both verbally and non-expressed both verbally and non-verbally. Emotional speech is verbally. Emotional speech is usually coupled with facial usually coupled with facial expressions and may also include expressions and may also include body gestures. body gestures.


Emotion involves our entire being, Emotion involves our entire being, the brain , the heart beat rate, the brain , the heart beat rate, blood pressure, muscle blood pressure, muscle movements, etc.movements, etc.

Emotions can be pleasant or Emotions can be pleasant or unpleasant.unpleasant.


Our Our QuestsQuests

How do we express emotions in How do we express emotions in speech in a tonal language such speech in a tonal language such as Thai?as Thai?

Other than words, what are the Other than words, what are the features in speech that are cues features in speech that are cues for emotions expressed by the for emotions expressed by the speaker and taken up by the speaker and taken up by the listener?listener?

Are these acoustic cues universal Are these acoustic cues universal or language specific?or language specific?


AssumptioAssumptionsns

Intonation is the cue for Intonation is the cue for emotional speech.emotional speech.

There is uniqueness in an There is uniqueness in an overall pattern of intonation for overall pattern of intonation for each type of emotional speech. each type of emotional speech.


InvestigationsInvestigations

Four Four types of emotionstypes of emotions most most commonly found in 32 languages; commonly found in 32 languages; AngerAnger, , SurpriseSurprise, , HappinessHappiness, , Sadness.Sadness.

Subjects:Subjects: two radio performers, two radio performers, male and female aged 30-40 years.male and female aged 30-40 years.


Test tokens:Test tokens: 4 one-word and 3 4 one-word and 3 multiple- word utterances all said multiple- word utterances all said in neutral and in four emotions, in neutral and in four emotions, five times each, a total of 350 five times each, a total of 350 (7x5x5x2) utterances were (7x5x5x2) utterances were investigated.investigated.


The acoustic measurments and The acoustic measurments and calculations made for each type of calculations made for each type of utterance in each type of emotionutterance in each type of emotion::

- average Fo (Hertz),- average Fo (Hertz),

- Fo range (semitone),- Fo range (semitone),

- maximum, minimum amplitude - maximum, minimum amplitude (dB),(dB),

- average amplitude (dB), - average amplitude (dB),

- utterance duration (second),- utterance duration (second),

- speaking rate (utterance/second)- speaking rate (utterance/second)


Figure 1: Stylized Fundamental Frequency (in Hertz) and neutralized duration (in Percentage) for one-word utterances (a proper noun CVVN).

Female 1-word Utterances

100

150

200

250

300

350

400

0 10 20 30 40 50 60 70 80 90 100

Percentage (%)

F0

(Hz)

Neutral

Anger

Surprise

Happiness

Sadness


Figure 2: Stylized Fundamental Frequency (in Hertz) and neutralized duration (in Percentage) for one-word utterances (a proper noun CVVN).

Male 1-word Utterances

100

150

200

250

300

0 10 20 30 40 50 60 70 80 90 100

Percentage (%)

F0

(Hz)

Neutral

Anger

Surprise

Happiness

Sadness


Figure 3: Stylized Fundamental Frequency (in Hertz) and neutralized duration (in Percentage) for

three-word utterances “คุ�ณแนนมา”.

Female 3-word

150

200

250

300

350

400

450

0 10 20 30 40 50 60 70 80 90 100

Percentage (%)

F0

(Hz)

Neutral

Anger

Surprise

Happiness

Sadness


150

200

250

300

350

400

450

0 10 20 30 40 50 60 70 80 90 100

Percentage (%)

F0

(Hz)

Neutral

Anger

Surprise

Happiness

Sadness



three-word utterances “คุ�ณแนนมา”.


100

150

200

250

300

0 10 20 30 40 50 60 70 80 90 100

Percentage (%)

F0

(Hz)

Neutral

Anger

Surprise

Happiness

Sadness


Results:Results: Average Utterance Average Utterance DurationDuration

Duration Duration NeutraNeutral l

Anger Anger SurprisSurprise e

HappineHappiness ss

SadnesSadness s

Average

0.59 0.52 0.49 0.73 0.61

S.D. 0.36

(n =70)

0.35

(n = 70)

0.32

(n =70)

0.35

(n=70)

0.42

(n=70)Table 1: Average utterance duration (in second) for all types of utterances combined in neutral speech and in each emotional type; anger, surprise, happiness, and sadness. (male and female combined)


Average Duration

0.000.100.200.300.400.500.600.700.80

1

Emotions

Tim

e (S

ec)

Neutral Anger Surprise Happiness Sadness

0.590.52 0.49

0.730.61

Figure 5:: Average utterance duration (in second) for all types of utterances combined (male and female combined)


Speaking RateSpeaking Rate

happiness > sadness > neutral > anger > s urprise ------------ slowest > fastest speaking rate

(utterance/second)


Pitch RangePitch Range

Pitch Pitch RangeRange

NeutraNeutral l

Anger Anger SurprisSurprise e

HappineHappiness ss

SadnesSadness s

Average

5.40 7.11 5.48 7.49 5.49

S.D. 2.51

(n=70)

2.20

(n=70)

2.12

(n=70)

3.40

(n=70)

1.75

(n=70)Table 2: Average Pitch Range (in semi-tone) for all types

of utterances combined in neutral speech and in each emotional type; anger, surprise, happiness, and sadness. (male and female combined)


Average Pitch Range

012345678

1

Emotions

sem

i-to

ne


5.40

7.11

5.48

7.49

5.49

Figure 6: Average Pitch Range (in semi-tone) for all types of utterances combined in neutral speech and in each emotional type; anger, surprise, happiness, and sadness. (male and female combined)


Pitch RangePitch Range

happiness > anger > sadness/ surprise > neutral

-------------- largest > smallest pitch ran ge

(semitones)


Average PitchAverage Pitch

Table 3:Table 3: Average F0 (in Hertz)Average F0 (in Hertz) for one-word for one-word utterances (2 proper names and 2 verbs utterances (2 proper names and 2 verbs combined) and multiple-word utterances (3, 5 combined) and multiple-word utterances (3, 5 and 6 combined).and 6 combined). (male and female)(male and female)

SpeakSpeakerer

UtteranceUtterancess

NeutrNeutralal

AngerAnger SurprisSurprisee

HappineHappinessss

SadneSadnessss

mm -one word-one word 118.75118.75 210.35210.35 200.12200.12 208.70208.70 125.94125.94

-multiple w-multiple wordord

133.62133.62 228.27228.27 227.28227.28 226.72226.72 128.84128.84

ff -one word-one word 205.67205.67 260.04260.04 204.67204.67 189.74189.74 193.53193.53

-multiple w-multiple wordord

207.47207.47 235.72235.72 209.11209.11 217.22217.22 202.21202.21


Average PitchAverage Pitch

anger > surprise/happiness > neutral/sa dness

-------------- highest > lowest pitch (Hertz)


Maximum & Minimum Maximum & Minimum AmplitudeAmplitude

Table 4: Maximum and Minimum Amplitude (dB) for all utterances combined in all types of emotion. (male and female)

SpeakerSpeaker AmplituAmplitudede

NeutraNeutrall

AngeAngerr

SurprisSurprisee

HappinesHappinesss

SadnesSadnesss

MaleMale MaximuMaximumm

72.84 83.60 75.46 79.07 71.54

MinimumMinimum 46.51 50.78 50.45 44.47 43.10

FemaleFemale MaximuMaximumm

67.30 75.85 72.97 70.73 67.80

MinimumMinimum 35.93 46.55 42.42 38.79 33.14


Figure 7: Maximum and Minimum Amplitude (dB) for all utterances combined in all types of emotion. (male and female)

Maximum and Minimum Amplitude (dB)

-60

-40

-20

0

20

40

60

80

100

Emotions

Male

Female


Max

Min

72.84

83.60

67.30

75.85

75.46

72.97

79.07 70.

7371.54

67.80

46.51

50.78

35.93 46.

5550.45

42.42

44.47

38.79

43.10

33.14


Average AmplitudeAverage Amplitude

Table 5: Average Amplitude (dB) for all utterances combined in each type of emotions. (male and female)

SpeakerSpeaker AmplituAmplitudede

NeutraNeutrall

AngerAnger SurprisSurprisee

HappineHappinessss

SadnesSadnesss

MaleMale XX 62.32 72.40 65.87 67.24 58.71

S.D.S.D. 2.04

(n=35)

3.10

(n=35)

2.70

(n=35)

3.87

(n=35)

3.16

(n=35)

FemaleFemale XX 57.80 66.58 62.89 61.88 54.19

S.D.S.D. 1.82

(n=35)

3.70

(n=35)

3.09

(n=35)

2.72

(n=35)

4.08

(n=35)


Figure 8: Average Amplitude (dB) for all utterances combined in each type of emotions. (male and female)

Average Amplitude

01020304050607080


Emotions

dB Male

Female

62.32

57.80

72.40

66.58

65.87 62.

89

67.2461.8

858.71 54.

19


Average AmplitudeAverage Amplitude

anger > happiness/surprise > neutral > sadness

----------- highest > lowest amplitude (dB)


DiscussionDiscussion

Luksaneeyanawin (1983)Luksaneeyanawin (1983) in her studies in her studies of Thai intonation found that in of Thai intonation found that in expressing expressing anger, the average pitch is anger, the average pitch is either higher (or lower) than neutral either higher (or lower) than neutral speech, with a wider pitch range, speech, with a wider pitch range, either longer (or shorter) duration and either longer (or shorter) duration and a very high degree of loudnessa very high degree of loudness. For . For surprise, the average pitch is higher surprise, the average pitch is higher than neutral speech, with a narrower than neutral speech, with a narrower pitch range, either shorter (or longer) pitch range, either shorter (or longer) duration and a higher (or lower) duration and a higher (or lower) degree of loudnessdegree of loudness..


Cahn (1988)Cahn (1988) in her studies of in her studies of emotions expressed in emotions expressed in English English intonationintonation found that found that anger is anger is expressed in a faster and louder expressed in a faster and louder speech with a larger pitch range and speech with a larger pitch range and higher average pitch than in neutral higher average pitch than in neutral speech. The pitch contour fluctuates speech. The pitch contour fluctuates more with greatest energy found in more with greatest energy found in higher frequencies.higher frequencies. And for And for grief, the grief, the speech is found to be slow with low speech is found to be slow with low pitch and weak high frequencies.pitch and weak high frequencies.


The data found in our two Thai The data found in our two Thai speakers as shown in tables 1-5 and speakers as shown in tables 1-5 and figures 1-8 above are, more or less, figures 1-8 above are, more or less, in agreement with in agreement with Luksaneeyanawin (1983) for Luksaneeyanawin (1983) for angeranger and and surprisesurprise.. Our data are also Our data are also in in agreement with Cahn (1988) foragreement with Cahn (1988) for angeranger andand sadnesssadness.. (Happiness is (Happiness is yet to be compared).yet to be compared).


In sum, for our two Thai speakers,In sum, for our two Thai speakers,

Anger Anger is expressed with highest is expressed with highest amplitude, highest average pitch, amplitude, highest average pitch, larger pitch range, and faster larger pitch range, and faster speaking rate when compared with speaking rate when compared with other emotional types and neutral other emotional types and neutral speech. speech.

Conclusion


SurpriseSurprise is expressed with a higher is expressed with a higher average pitch, higher amplitude, average pitch, higher amplitude, and shorter duration than neutral and shorter duration than neutral speech. Our data show a smaller speech. Our data show a smaller pitch range when compared with pitch range when compared with other emotional types but larger other emotional types but larger than neutral speech for surprise. than neutral speech for surprise.


SadnessSadness is expressed with lowest is expressed with lowest average pitch, lowest amplitude, average pitch, lowest amplitude, medium average pitch range, and medium average pitch range, and slower speaking rate when slower speaking rate when compared with other types of compared with other types of emotion. emotion.

The average pitch for sadness is The average pitch for sadness is comparable to that of neutral comparable to that of neutral speech in Thai.speech in Thai.


HappinessHappiness is found to be expressed is found to be expressed withwith slowest speaking rate, slowest speaking rate, largest pitch range, higher largest pitch range, higher average pitch and higher average pitch and higher amplitude than neutral speech in amplitude than neutral speech in our Thai speakers. This is yet to our Thai speakers. This is yet to be compared with other data.be compared with other data.


For For Fo contourFo contour, there is , there is no no uniformity in the present datauniformity in the present data for each type of the emotional for each type of the emotional speech observed in either male speech observed in either male or female speaker.or female speaker.


Unlike the Fo contour of neutral Unlike the Fo contour of neutral declarative statements declarative statements (Tumtavitikul and Thitinarnnara (Tumtavitikul and Thitinarnnara 2006) where the Highs and Lows 2006) where the Highs and Lows conform to the universal tendency conform to the universal tendency of Fo declination in declarative of Fo declination in declarative sentences (Hirst and Di Crito 1988) sentences (Hirst and Di Crito 1988) and the Highs and Lows are clearly and the Highs and Lows are clearly rule-based even with focus shifted.rule-based even with focus shifted.


The Highs and Lows in the The Highs and Lows in the present data dopresent data do not form a unified not form a unified pattern in any type of the four pattern in any type of the four emotions observed. The shapes emotions observed. The shapes of the contour vary greatly among of the contour vary greatly among the same type of emotion in both the same type of emotion in both male and female speakers. male and female speakers.


This is not surprising since This is not surprising since emotions are subject to mood and emotions are subject to mood and attitude as well as environments. attitude as well as environments. Emotional pitch contours may vary Emotional pitch contours may vary greatly within and among greatly within and among individuals (Ladefoged 2006).individuals (Ladefoged 2006).


It may be induced from the comparable It may be induced from the comparable features found in Luksaneeyanawin, our Thai features found in Luksaneeyanawin, our Thai and Cahn’s English data for the three types and Cahn’s English data for the three types of emotional speech, of emotional speech, angeranger, , surprisesurprise and and sadnesssadness, that , that emotional speechemotional speech may have a may have a universal tendency of expression.universal tendency of expression. The The unifomity of expression may not be unifomity of expression may not be found in the intonation contour per sefound in the intonation contour per se but in other acoustic cues e.g., but in other acoustic cues e.g., speaking rate, F0 range, average F0, speaking rate, F0 range, average F0, amplitude contour.amplitude contour.


This is readily This is readily explainable since emotions explainable since emotions which affect humans which affect humans mentally and psychologically mentally and psychologically are impulses or natural are impulses or natural

responses to stimuliresponses to stimuli..


Ververidis, Kotropoulos and Pitas Ververidis, Kotropoulos and Pitas (2004)(2004) in their studies of in their studies of automatic classification of automatic classification of emotional speech in Danish five emotional speech in Danish five emotional states; emotional states; angeranger, , happinesshappiness, neutral, , neutral, sadnesssadness, and , and surprisesurprise, reported accuracy rates , reported accuracy rates of classification between 20-52%. of classification between 20-52%.

ImplicationsImplications


The set of features used for The set of features used for speech generations in Ververidis, speech generations in Ververidis, Kotropoulos and Pitas (2004)Kotropoulos and Pitas (2004) were mainly abstracted from the were mainly abstracted from the pitch and energy contourspitch and energy contours of of natural speech.natural speech.


Vroomen, Collier and Mozziconacci Vroomen, Collier and Mozziconacci (1993) in their acoustics studies of (1993) in their acoustics studies of intonation in emotional speech in Dutch intonation in emotional speech in Dutch found that found that intonation and duration are intonation and duration are sufficient to express emotionssufficient to express emotions. By re-. By re-synthsizing the intonation contour copied synthsizing the intonation contour copied from natural speech, with proper time from natural speech, with proper time alignments, onto monotonous utterances. alignments, onto monotonous utterances. The rule-based manipulations of pitch The rule-based manipulations of pitch and duration were found to suffice in and duration were found to suffice in representing emotionsrepresenting emotions. .


The parametersThe parameters in Vroomen, in Vroomen, Collier and Mozziconacci (1993) Collier and Mozziconacci (1993) for their synthesis were the type for their synthesis were the type of emotion targeted, excursion of emotion targeted, excursion size, the key in the register, and size, the key in the register, and the optimal duration of the the optimal duration of the utterancesutterances. .


Our findings may have Our findings may have implications for Thai speech implications for Thai speech technology. Utterances can be technology. Utterances can be manipulated in the dimensions of manipulated in the dimensions of time, average fundamental time, average fundamental frequency and fundamental frequency and fundamental frequency range as well as frequency range as well as amplitude to represent amplitude to represent emotional speech according to emotional speech according to the type of emotion targeted. the type of emotion targeted.


Conversely, these features can be Conversely, these features can be used as parameters for automatic used as parameters for automatic recognition/classification of recognition/classification of emotional speech. emotional speech. Further Further studies are necessary to derive studies are necessary to derive the rules governing the the rules governing the alignments of these parameters alignments of these parameters with the utterances.with the utterances.


AngerAnger

Source: Emoticon for Angry (www.danwade.com)> ; - (

http://smiley.smileycentral.com/download


SurpriseSurprise

Source: Emoticon for Gasp (www.danwade.com)

= - 0



HappinessHappiness

Source: Emoticon for Smile (www.danwade.com): - )



SadnessSadness

Our suggested emotion symbol for sadness

<:-( or :-(



AcknowledgementAcknowledgement

This paper is a part of the This paper is a part of the research project, Thai Intonation in research project, Thai Intonation in Different Emotions, funded by Different Emotions, funded by Kasetsat University Research and Kasetsat University Research and Development Institute grant no. Development Institute grant no.

สส--ข ข ((มนมน..)) 1.47 to Apiluck 1.47 to Apiluck TumtavitikulTumtavitikul


ReferencesReferences

Cahn, Janet. 1988. From Sad to Glad: Emotional Computer Cahn, Janet. 1988. From Sad to Glad: Emotional Computer Voices.Voices. Proceedings of Speech Tech '88, Voice Input/ Proceedings of Speech Tech '88, Voice Input/ Ouput Ouput Applications Conference and ExhibitionApplications Conference and Exhibition. New York City. . New York City. April, 1988. Pages 35- 37. April, 1988. Pages 35- 37.

Dimitrios Ververidis, Constantine Kotropoulos, and Ioannis Pitas. Dimitrios Ververidis, Constantine Kotropoulos, and Ioannis Pitas. 2004.2004. Automatic Emotional Speech Classification. Automatic Emotional Speech Classification.

Proceedings Proceedings of the International Conference on Acoustics, of the International Conference on Acoustics, Speech andSpeech and Signal Processing (ICASSP) 2004Signal Processing (ICASSP) 2004. Montreal, Canada.. Montreal, Canada.

Hirst, D. and Di Cristo, A. 1998. A Survey of Intonation Hirst, D. and Di Cristo, A. 1998. A Survey of Intonation Systems. Systems. Intonation SystemsIntonation Systems, ed. by D. Hirst and A. Di Cristo. , ed. by D. Hirst and A. Di Cristo.

Cambridge. Camgridge University.Cambridge. Camgridge University.

Kanlayarat, Thitikannara. 2006. Kanlayarat, Thitikannara. 2006. Thai Intonation in Different Thai Intonation in Different EmotionsEmotions. Unpublished MA Thesis, Kasetsart . Unpublished MA Thesis, Kasetsart

University. University.


Ladfoged, Peter. 2006. Ladfoged, Peter. 2006. A Course in Phonetics, A Course in Phonetics, 5th ed. 5th ed. Boston: Thomson Wadsworth.Boston: Thomson Wadsworth.

Luksaneeyanawin, Sudaporn. 1983. Luksaneeyanawin, Sudaporn. 1983. Intonation in ThaiIntonation in Thai. . Unpublished Doctoral Dissertation, University ofUnpublished Doctoral Dissertation, University of

Edinburgh.Edinburgh.

Tumtavitikul, Apiluck and Thitikannara, Kanlayarat. 2006. Tumtavitikul, Apiluck and Thitikannara, Kanlayarat. 2006. Stress and Intontaion in Thai. Stress and Intontaion in Thai. Journal of Language Journal of Language

and and Linguistics Linguistics 24.2.59-7624.2.59-76. .

Ververidis, Dimitrios and Kotropoulos, Constantine. 2003. Ververidis, Dimitrios and Kotropoulos, Constantine. 2003. A A

State of the Art Review on Emotional Speech State of the Art Review on Emotional Speech DatabasesDatabases. (http://poseidon.dsd.auth.gr/ EN/index.html). (http://poseidon.dsd.auth.gr/ EN/index.html)

Vroomen, Jean, Collier, Rene, and Sylie Mozziconacci. Vroomen, Jean, Collier, Rene, and Sylie Mozziconacci. 1993. 1993. Duration and Intonation in Emotional Speech. Duration and Intonation in Emotional Speech. Proceedings of Eurospeech 1993Proceedings of Eurospeech 1993 (pp.577–580 (pp.577–580). Berlin, ). Berlin, Germany.Germany.

Thai Intonation in Four Thai Intonation in Four EmotionsEmotions

Apiluck Tumtavitikul Apiluck Tumtavitikul andand Kanlayarat Kanlayarat ThitikannaraThitikannara

Linguistics Dept., Kasetsart UniversityLinguistics Dept., Kasetsart UniversityBangkok, Thailand 10900Bangkok, Thailand 10900email: email: [email protected]@ku.ac.th

http://pirun.ku.ac.th/~fhumalthttp://pirun.ku.ac.th/~fhumalt



five-word utterances “คุ�ณแนนมาหาคุ�ณ”. .


150

200

250

300

350

400

450

0 10 20 30 40 50 60 70 80 90 100

Percentage (%)

F0

(Hz)

Neutral

Anger

Surprise

Happiness

Sadness



five-word utterances “คุ�ณแนนโทรมาหาคุ�ณ”.


100

150

200

250

300

0 10 20 30 40 50 60 70 80 90 100

Percentage (%)

F0

(Hz)

Neutral

Anger

Surprise

Happiness

Sadness



six-word utterances “คุ�ณแนนโทรมาหาคุ�ณ”.


100

150

200

250

300

350

0 10 20 30 40 50 60 70 80 90 100

Percentage (%)

F0

(Hz)

Neutral

Anger

Surprise

Happiness

Sadness



six-word utterances “คุ�ณแนนโทรมาหาคุ�ณ”.


150

200

250

300

350

400

450

0 10 20 30 40 50 60 70 80 90 100

Percentage (%)

F0

(Hz)

Neutral

Anger

Surprise

Happiness

Sadness

Thai Intonation in Four Emotions Apiluck Tumtavitikul and Kanlayarat Thitikannara Linguistics Dept.,...

Documents

Transcript of Thai Intonation in Four Emotions Apiluck Tumtavitikul and Kanlayarat Thitikannara Linguistics Dept.,...