3.0 Speech Synthesis
Imperial Academy of St Petersberg put the following questions as the subject of their annual prize.
1. What is the nature and character of the sounds of the vowels a, e, i, o, u that make them so different from one another?
2. Can an instrument be constructed like the vox humana pipes of an organ which shall accurately express the sounds of the vowels.
Winner was Kratzenstein 1779
A E I O U
Von Kempelen’s speaking machine (1791)(Wheatstone’s reconstruction
Voder (Voice Operation Demonstrator) 1939
3.0 Speech Synthesis
The emphasis these days is very much on text-to-speech(TTS) synthesis and we shall concentrate on this method.
One can consider TTS systems as consisting essentially of two components – a high-level component and a low-level component – or perhaps as a front end and back end.
Current speech synthesis systems may be classified, depending largely on their low-level component, as
formant parameter systems
waveform concatenation systems
Audio clips of synthetic speech illustratingthe history of the art and technology of synthetically produced human speech.
http://www.cs.indiana.edu/rhythmsp/ASA/Contents.html
http://www.cs.indiana.edu/rhythmsp/ASA/highlights.html
http://www.humnet.ucla.edu/humnet/linguistics/faciliti/demos/vocalfolds/vocalfolds.htm
Human speech
Parametric analysis
Normalisation
SPRUCE Text-to-Speech Synthesis
Build syllable sentence
Parse sentence
Perform prosodic processing
High level output
Text input
DictionaryRules
Pragmatics
Retrieve andconjoin syllables
Assemble output file
Holmes synthesiser
Retrieve andconjoin syllables
Assemble output file
DAC
Syllable inventoryInventory
ofwaveform
units
This may also be referred to as synthesis-by-rule although rules of one sort or another are common to all synthesis systems. For a formant synthesis system the output of the high-level component typically consists of a sequence of allophones together with their duration and pitch, e.g.
DH 7 34 I 5 34 S 8 33
Formant Synthesis
Duration measured in 10ms frames
Pitch coded into the range 1-63
The low-level component uses this input to provide a sequence of frames, each frame containing a set of parameters referring to formant frequencies, formant amplitudes, voicing, fundamental pitch, etc., e.g.
Fn alf f1 a1 f2 a2 f3 a3 ahf s f0 250 33 280 33 1300 32 2680 34 41 36 34 250 37 280 37 1300 36 2680 38 45 36 34 250 40 280 40 1300 39 2680 41 48 36 34 250 42 280 42 1300 41 2680 43 50 36 34 This information is then fed into a formant synthesiser which uses it to generate the appropriate audio output. The formant synthesiser may be implemented in hardware or software.
An example of a formant synthesis system is DECTALK.
RZ R1 R2 R3 R4RN
R1
R2
R3
R4
R5
R6
Glottal filter
Impulse train
Random numbers
LP filter
A1
A2
A3
A4
A5
A6
Pre-emphasis+
Parallel
R5
Synthetic speech output
Cascade
Klatt Synthesiser
Klatt synthesiser
A combined serial/parallel formant synthesiser.
A serial, or cascade, synthesiser is a better model for the production of vowel and vowel-like sounds whereas a parallel synthesiser is better suited to producing nasals, fricatives and stops.
The serial synthesiser specifies the formant centre frequencies and bandwidths.
The parallel synthesiser specifies the formant levels (peak amplitudes) also.
Waveform Concatenation Synthesis
With this system the low-level component generates a speech output file by concatenating units of previously recorded speech. Informationabout the duration and pitch of these units is again supplied by the high-level component. The size of the units is clearly an important consideration and both in terms of amount of storage required and the difficulties involved with joining them together (more about this later).
An example of a waveform concatenation synthesis system is the Lernout and Hauspie TTS system.
cer- -tain- -ly(ser-) (-tan-) (-lee)
What are the options for stored units?
Sentences
Phrases
Words
Syllables
Phonemes
Demi-syllables
Diphones
Stored Speech Units
words
How many? ~ 300,000Perhaps 2000 for most frequently used words.
Formant synthesis1 word ~ 0.5sec12bytes per 10msec frame1 word = 0.5/.010 frames = 50 frames = 600 bytes
300,000 words ~180Mbyte
Waveform concatenation1 word ~ 0.5 secSampling rate = 16Khz1 word = 8K samples = 16K bytes at 2 bytes per sample
300,000 words = 4.8Gbyte
morphemesBasic meaningful units that make up words, essentially roots, prefices, sufficese.g. sail, travel, -ed, -s => sail, travel sailed, travelled sails, travels ~10,000-30,000 entries
Formant synthesis30,000 entries ~ 10-15Mbyte
Waveform concatenation30,000 entries ~ 120Mbyte
syllables
How many? ~ 10,000??
Formant synthesis1 syllable ~ 0.3sec12bytes per 10msec frame1 syllable = 0.3/.010 frames = 30 frames = 360 bytes
10,000 syllables ~3.6Mbyte
Waveform concatenation1 syllable ~ 0.3 secSampling rate = 16Khz1 syllable = 4.8K samples = 9.6K bytes at 2 bytes per sample
10,000 syllables = 96Mbyte
phonemes
Formant synthesis.
1 phoneme ~ 10 frames = 120 bytes
40 phonemes ~5Kbytes
Actually need about 70-80 allophones,giving ~ 10Kbytes
How many? About 40 for English.
Waveform concatenation
1 phoneme ~ 100msec = 0.1sec = 1.6K samples = 3.2K bytes
Total ~ 256K
demi-syllables
s u m t ie m z (sometimes)
These are units of speech obtained by making cuts in the middle of the vowel part of the syllable.The reason for doing this is that coarticulation effects are minimal in the middle of the vowel.The number of demi-syllables is about 4-5000 made up from about 1500 initial demisyllables and 3000 final demisyllables.
s u m - t ie m z
cer- -tain- -ly(ser-) (-tan-) (-lee)
Demi-syllables
diphones
k u n uu (canoe)
These are units of speech obtained by going from the middle of onephone to the middle of another.The reason for doing this is that coarticulation effects are minimal in the middle of the sound.Theoretically there are about 40x40 = 100 diphones but in practice thenumber is about 1200.
qk | ku | un | nuu | uuq
q = silence
cer- -tain- -ly(ser-) (-tan-) (-lee)
Diphones
Words versus phonemes
Trade off between storage and processing
The larger the unit then the more storage space it requiresbut in compensation less effort is required in joining the units together.
Pronunciation Task
Input Text
AllophoneTask
Allophonic Rules
Phonemic Text
Broad Phonetic Text
Affix Tables
Pronunciation Rules
Restricted Text
Phonemic Text
Conversion Task
Exceptions Dictionary
Restricted Text
ProsodyTask
Prosody Table
Broad Phonetic Text
Narrow Phonetic Representation
Lower Phonetic Task Lower Phonetic
Table
Narrow Phonetic Representation
Control Parameters
Phonotactic Tables
Speech
Conversion Task
Pronunciation Task
Prosody Task
Lower Phonetic Task
Input textExceptions Dictionary
Phonotactic Tables
Affix Tables
Pronunciation Rules
Prosody Table
Lower Phonetic Table
The JSRU TTS SystemSpeech Output
This task converts unrestricted text to restricted text. Unrestricted text consists of non-English words, abbreviations (e.g. Dr.), unusual pronunciation, words to be spelt (e.g. BBC), etc. This task also gets the phonetic form of words that are in the dictionary and deletes any redundant white space.
The date is 1/10/97. …the first of October, ninety ninety seven.
I can’t do it. …I cant do it. Or I cannot do it
The price is £23.99. … twenty three pounds ninety nine pence
St. George St. …Saint George Street
well-behaved …well behaved
CONVERSION TASK
PRONUNCIATION TASK
1. Look up entire word in the dictionary.If found then exit from pronunciation task.
2. Remove suffixes.After removing a suffix look up remainder of word in the dictionary. If found go to 5 else try to remove a further suffix.Continue until no further suffixes can be removed.
3. Remove prefixes.
4. Perform pronunciation rules on the stem.
5. Replace prefixes and suffixes in their phonetic form, of course,obtained from lookup tables.
6. Perform stress assignment by applying the stress rules.
7. Divide the word into syllables.
1. Look up the entire word in the dictionary. If found then exit from pronunciation task.
sentence “sen-tanssensitive “sen’si-tivtransport “traans-poatcough “kof
Conversion of words to phonemes is far from regular and is very context dependent in English. Some languages are better than others in this respect.
George Bernard Shaw is quoted as saying that fish is spelt GHOTI??
The advantage of using a dictionary is that it can include information on syllables, stress, syntactic types, etc. This is much more easily obtained using a lookup table than by using algorithms. For example, stress markers and syllable boundaries are more easily identified. But, of course, there will always be some words which are not in the dictionary, however large one makes it.
2. Remove suffixesAfter removing a suffix look up the remainder of the word in the dictionary. If found then go to 5 else try to remove a further suffix. Continue until no further suffixes can be removed.
e.g. wonderfully -> wond - er - ful – ly
First remove –s
parties -> party -s
The ‘e’ needs to be deleted for words like dishes because otherwise the final ‘e’ lengthens the previous vowel (magic ‘e’ rule). e.g. fades -> fade -s
fishes -> fish -s
All other suffixes treated consistently.
Example of function for -s removal (not definitive)
lastchar = Sif (prevchar=S) then
returnelse if (prevtwochars = IO) then
remove Selse if (prevchar=vowel but not E) then
returnelse if (prevchar=E) then
remove Sif (prevprevchar=I) then
replace IE by Yelse if (prevprevchar=H) and (prevprevprevchar T) then
delete Eelse if ((prevprevchar=S) and (prevprevprevchar=S))
or ((prevprevchar=Z) and (prevprevprevchar=Z)) thendelete E
SS
IOS
IES -> Y
HES but not THES
SSES/ZZES
loss
folios
parties
batches/bathes
losses/buzzes
AS alias
A table of suffixes contains for each suffix: its spelling a list of letters that may precede a list of suffixes which may precede an indication of whether it can replace a final 'e' its pronunciation its stress class For example, "ICAL" "" anything can precede it "ENT possible preceding suffixes 0 cannot replace a final 'e' "ikal" phonetic spelling -2 mainstress placed two syllables to the
left of the last syllable in the suffix
e.g. radical stress on 'rad' identical stress on ‘ent'
The JSRU system has a list of 39 suffixes e.g. ED,NT, FUL, OUS, ALIC, IBLE, EN, etc.
After a suffix has been removed, the stem is checked to see if it is long enough(at least 3 letters) and whether the final consonant cluster is pronounceable.
If the suffix could have replaced a final 'e' then this 'e' must be added. The algorithm for deciding whether or not an 'e' has been removed is not simple.e.g.alternation -> alternate + ioninteraction -> interact + ion
Rule here is that if stem ends in vowel + consonant then an 'e' should be added. What about taxable? OK when 'x' replaced by 'ks'.Having removed a suffix, the reduced word is looked up in the dictionary. If it’s not found then an attempt is made to remove another suffix.
e.g. wond - er - ful - ly
3. Remove prefixes in a similar fashion to the suffixes.
Similar procedure to removal of suffixes but with simplification that a prefix doesn't alter the stem.Table of prefixes contains for each prefix: its spelling list of letters which may follow it list of prefixes which may follow it its pronunciation stress placement control For example,"AL" *aeioul none of a, e, i, o, u, l "" any"awl" phonetic spelling0 prefix may be stressed
e.g. always, already
4. After removal of suffixes and affixes we are left with the stem of the word.
Perform pronunciation rules on the stem.
Pronunciation of the stem is worked out by a set of rules, unless it's found in the dictionary.
Three sets of rules
The first set consists of 105 rules, each one contains
string to be changed to phonetic form
left context
right context
replacement string
For example,
{"A", "", "consY>", "ai"} lazy
{"OUGH", "", "", "ou"} bough
{"AUGH", "", "", "aw"} daughter
{"C", "", "E", "S"} lace
{"PH", "", "", "F"} phase
What about laughter?
What about rough, cough, though?
The order of the rules is important. e.g.
Special procedures need to be called in some cases:
{"Q", "", "", "KW"} freeqent freekwent
{"E", "cons", "Q", "EE"} freqent freeqent
{"QU", "", "", "Q"} frequent freqent
N.B. Every rule is tried at every position in the word. Hence, time taken
depends very much on the number of rules.
{"vowel", "", ""consE", "@001"} Magic 'e' procedure
{"E", "", ">", "@002} final 'e' procedure
{"@003", "", "", ""} double letter procedure
5. Replace the prefixes and suffixes – in their phonetic form, of course, which can be obtained from lookup tables. This may result in some adjustments which are handled by the second set of rules. e.g. {"c", "", "i", "s"} precious
{"c", "", "", "k"} practically
{"ig", "", "n-", "ie"} assignment
6. Perform stress assignment by applying the stress rules – these can be quite complicated. Make adjustments to final pronunciation e.g. reduction of some unstressed vowels. The stress rules are based on MIT rules and are quite complicated. (From text to speech: the MITalk system Allen et al)
In the first (cyclic) phase several rules are applied in sequence first to the stem then to the stem + suffixes taken one at a time Prefixes are considered as part of the stem Some affixes(e.g. -ING) do not affect the stress pattern so rules are omitted, while others(e.g. -ION) force stress onto a particular vowel, usually the one before the affix.
In the second (non-cyclic) phase one vowel is selected for mainstress and all others are reduced to secondary or no stress.
Stress Rules
Main Stress Rule (cyclic)
1. V -> [1-stress] / X – C0 {[short v] C01 / V} {[short V] C0 / V}
where X contains all prefixes and the symbol ‘-’ indicates the position of the vowel to be stressed C0
1 matches zero or one consonant C0 matches any number of consonants (including none) {..} denotes a list of alternative patterns separated by slashes /.
Assign 1-stress (primary stress) to the vowel in a syllable which precedes a weak syllable followed by a morph-final syllable containing a short vowel and zero or more consonants.
e.g. difficult - > d ” i f i k a l t X – C0 {..} {…}
V -> [1-stress] / X – C0 {[short v] C01 / V} {[short V] C0 / V}
where X contains all prefixes and the symbol ‘-’ indicates the position of the vowel to be stressed
Assign 1-stress to the vowel in a syllable preceding a vowel followed by a morph-final syllable containing a short vowel and zero or more consonants.e.g. secretariat -> sekret“eir ee aat X – C0 {..} {..}
Assign 1-stress to the vowel in a syllable preceding a vowel followedby a morph-final vowel.e.g. oratorio -> orat“oar ee oa X – C0 {..} {..}
2. V -> [1-stress] / X – C0 {[short V] C0 / V} where X contains all prefixes
Assign 1-stress to the vowel in a syllable preceding a short vowel and zero or more consonants
e.g. edit -> “ed it bitumen -> bity”uum en agenda -> aj”en da
3. V -> [1-stress] / X – C0
where X contains all prefixes
Assign 1-stress to the vowel in the last syllable
e.g. stand -> st”aand parole -> paar”oal
Stressed Syllable Rule (cyclic)
Alternating Stress Rule (cyclic)
Destressing Rule (non-cyclic)
Compound Stress Rule (non-cyclic)
Strong First Syllable Rule (non-cyclic)
Cursory Rule (non-cyclic)
Vowel Reduction Rule (non-cyclic)
After the stress assignment a third set of rules is applied. e.g.{"\141r", "", "cons", "\141"} far gone versus far away (141 = 'schwa'){"ir", "", "cons", "er"} dirt versus direct
and some unstressed vowels are reduced.e.g. aa aai i u a o a bottom => "bot-am
7. Divide the word into syllables. The rule used in the JSRU for this is pretty basic – not based on any principles of morphology. It simply places syllable boundaries half way between vowels.
e.g.
frequent -> freekwent -> freek – went
boundaries -> bound - reez
This task adds intonation and timing to the phonetic text. The task processes complete breath groups.
The output consists of a list of phonemes with corresponding pitch and duration.
PROSODY TASK
In November the region’s weather was unusually dry.
Do you want to travel to London?
Who is the Prime Minister of the Bahamas?
Lift the safety cover and press the red button.
Stress – two kinds•word stress•sentence stress
Word stressOne syllable in a polysyllabic word is given more prominence than the others.All words have one syllable with primary stress.
electricity university
tomorrow
controversy icecream
electricityuniversitytomorrow
controversy
icecream (or icecream?)
Examples
What time is the next train to London?
What time is the next train to London?
What time is the next train to London?
There’s John cycling down the road.
Sentence stressThis refers to the way in which one word is singled out in a sentence as the focus, or nuclear stress
Rhythm
Rhythm is about how we time the delivery of the sentence.
In English we are supposed to time the stressed syllables so thatthey occur equidistantly in time.
English is said to be a stress-timed language
The stressed syll–a-bles-in Eng-lish o - ccur e - qui- dis-tant-ly in time
# # # # # #
# # # # # # # # # # # # # # # #
clo pe die se lec tro niquescysenLe sontpleinesD’in for ma tions
Les encyclopedies electroniques sont pleines d’informations
French is said to be a syllable-timed language D Crystal 1995
•Set up the tone group boundaries.•Calculate durations of phonemes•Expand phonemes into lower phonetic segments•Set the pitch contour
Tone group boundaries are marked by a content word to the left and a function word to the right - provided this does not create a tone group without a content word.
JSRU PROSODY TASK
Utterance
Tone group 1 Tone group 2 Tone group N…….
/u u u / S u u S u u ……/ Smax/ u S u/ u /HeadPre-head Nucleus Tail
Projects represent a substantial part of your marks and you should spend some time choosing a project which will show you in your best light.
Projects represent // a substantial part // of your marks // and you should spend // some time choosing // a project // which will show // you in your best light.
“Pro-jects rep-res-”ent // a sub-”stant-ial “part // of your “marks // and you should “spend // some “time “choos-ing // a “pro-ject // which will “show // you in your “best “light.
Unstressed syllable weight 1Secondary stress “ 2Primary stress “ 3Emphatic stress “ 4
Tone groups are amalgamated such that all tone groups have a weight >=9.
“Pro-jects ‘rep-res-”ent // a ‘sub-”stant-ial “part // of your “marks //
and you should “spend // some “time “choos-ing // a “pro-ject
// which will “show // you in your “best “light.
Pitch contours are worked out for syllables according to their type, and the tone group type
Tone groups are classified as being of•Type 1 ends with a fullstop•Type 2 ends with a question mark•Type 3 ends with an exclamation mark•Type 4 any other tone group
10 10 5
6 8 5
5 9
See overheads!!!!
Baseline
When all syllable patterns have been calculated they are superimposed on a falling baseline
, ,
The value at the end of the phrase (indicated by the ,) is half that atthe beginning, and starting value for next phrase is 1.2 times the final value of previous one
TimingEach phoneme is represented by 1, 2 or 3 elements each of which hasan inherent duration and a minimum duration
Then actual duration = minimum + f * (inherent – minimum)
Rules for values of f (derived mainly by Klatt) are typically of the form
• For vocalic phonemes in a syllable which is not the last in a phrase multiply f by 0.6 (f initially set to 1)
• For phonemes with no stress or with secondary stress, multiply f by a factor as follows:
vocalic, word medial syllable 0.7other vocalic 0.85liquid or glide before a vowel 0.40all others 0.85
This task converts the output from PROSODY TASK into control parameters for the hardware synthesiser.
LOWER PHONETIC TASK
The table for each phonetic element contains information relating to howtransitions between target values are calculated. For each parameter in thetable there are entries for Its target value The proportion of the target of the dominated element used in deriving the boundary value A fixed contribution to the boundary value Transition duration within the dominant element Transition duration within the dominated element
Target Fixed
Contribution
Proportion Internal
Duration
External Duration
Rank
W 760 380 50 4 4 8
E 1900 950 50 3 3 2
It sends a frame of parameters to the synthesiser every 10msecs.
The boundary value is calculated as Fixed Contribution of Dominant element + Target of dominated element * proportion of dominant element i.e. 380 + 1900 * 0.5 = 1330
760 Hz
Target value for W
1330 Hz
1900 Hz
Target value for E
Internal duration4 frames
External duration4 frames
Target Fixed Contribution Proportion Internal Duration External Duration Rank
W 760 380 50 4 4 8
E 1900 950 50 3 3 2
Target Fixed Contribution
Proportion Internal Duration
External
Duration
Rank
DH 1300 650 50 2 5 6
A 1420 710 50 3 3 2
W 760 380 50 4 4 8
E 1900 950 50 3 3 2
L 880 440 50 0 6 11
Z 1720 860 50 2 5 6
DH A W E
1300
1360
1090
1330
1420
760
Boundary TargetValues Values
DH|A 650 + 1420 * 0.5 = 1360 DH 1330A|W 380 + 1420 * 0.5 = 1090 A 1420W|E 380 + 1900 * 0.5 = 1330 W 760
7 4 8 12
Tomorrow will be starting off grey and rather murky.
:"to-ma-rou /wil /bey :"star-ting /of :"grai /and :"raa-dha
:"mer-kee.
T 9 37TY 2 37TZ 3 37 O 10 37 M 9 35 A 4 35 R 4 34
OU 7 34OB 7 33 W 4 34 I 5 34
LP 4 32 B 5 33BY 1 33EY 5 33 S 11 36 T 6 35
TY 2 35AR 14 35 T 6 31TY 2 31 I 5 31NG 9 30 O 6 32 F 6 31 G 6 34GY 2 34 R 2 34AI 12 34AJ 9 32 A 4 32 N 6 31 D 2 30DY 1 30 R 7 33
AA 10 32DH 7 31 A 5 30 M 11 32ER 14 32 K 6 25KY 2 23 I 10 23QQ 51 22 Q 42 21
* Output from SYNTH: * fn alf f1 a1 f2 a2 f3 a3 ahf s f0* T 9 37 0 Frame 0 250 6 190 6 1780 6 2680 6 6 1 37 250 10 190 10 1780 10 2680 10 10 1 37 250 13 190 13 1780 13 2680 13 13 1 37 250 15 190 15 1780 15 2680 15 15 1 37 250 15 190 15 1780 15 2680 15 15 1 37 250 15 190 15 1780 15 2680 15 15 1 37 250 13 190 13 1780 13 2680 13 13 1 37 250 10 190 10 1780 10 2680 10 10 1 37 250 6 190 6 1780 6 2680 6 6 1 37* TY 2 37 0 Frame 9 250 15 190 15 1780 40 2680 51 58 1 37 250 15 190 15 1780 40 2680 51 58 1 37* TZ 3 37 0 Frame 11 250 6 190 6 1780 23 2680 28 40 1 37 250 10 190 10 1780 27 2680 32 44 1 37
BT1
Spruce 2
L&H 2
AT&T 2
Festival 2
BT 2
Mbrola 2
L&H 1
Spruce 1
Festival 1
AT&T 1
Mbrola 1
Top Related