Bootstrapping a Language-Bootstrapping a Language-Independent SynthesizerIndependent Synthesizer

Craig Olinsky

Media Lab Europe / University College Dublin

15 January 2002

Introducing the ProblemIntroducing the Problem

Given a set of recordings and transcriptions in an arbitrary language, can we quickly and easily build a speech synthesizer?

YES, if we know something about the language.

However, for the majority of languages for which such resources don’t exist…

PROS The existing synthesizer

provides a store of “linguistic” knowledge we can start from.

Analogue to speaker adaptation in Speech Recognition systems.

Overall, quality should be better.

CONS Difficulty related to degree of

different between sample and target language.

Best as a gradual process: accent/dialect, not language

Starting from SampleStarting from Sample

PROS Difficulty directly proportional to

complexity of the language.

Common (machine-learning) procedure based upon machine learning from recordings and transcript.

CONS Don’t have a great deal of

relevant knowledge to apply to the task.

If not using principled phone set, necessary to segment / label recordings cleanly

Starting from ScratchStarting from Scratch

The Obvious CompromiseThe Obvious Compromise

Take what we do know from building speech synthesis, and generalize it to an existing framework.

-- we’re not specifically learning from “scratch”

-- at the same time, we’re not making linguistic assumptions pre-coded into the source voices

““Generic” Synthesis Generic” Synthesis Framework/ToolkitFramework/Toolkit

Set of Scripts, Utilities, and Definition files to help to help to automate the creation of reasonable speech synthesis voices from an arbitrary language without the need for linguistic or language-specific information.

Build on top of the Festival Speech Synthesis System and FestVox toolkit (for wave form synthesis; most of text processing and pronunciation handling externalized to locally-developed tools)

Language-Dependent Language-Dependent Synthesis ComponentsSynthesis Components

Phone set

Word pronunciation (lexicon and/or letter-to-sound rules)

Token processing rules (numbers etc)

Durations

Intonation (accents and F0 contour)

Prosodic phrasing method

Phoneme SetsPhoneme Sets

If we rely on a pre-existing set of pronunciation rules, lexicon, etc., we are automatically limited to using the phone-set used in those resources (or something which they can be mapped to); most likely something language-dependent.

IPA, SAMPA: something language-universal?

We need to generate pronunciations: how do we create the relationship between our training database / phonetic representation / orthography?

““Multilingual” Phoneme Multilingual” Phoneme Sets: IPA, SAMPA Sets: IPA, SAMPA

We don’t want to be stuck with a set of phonemes targeted for a specific language, so we instead use a phoneme definition designed to be inclusive of all

But… this still assumes we know the relationship between the phone set and orthography of the language; i.e. for any given text we can generate a pronunciation.

This approach still assumes linguistic knowledge!

Orthography Orthography asas PronunciationPronunciation

cf: R. Singh, B. Raj and R.M. Stern, “Automatic Generation of Phone Sets and Lexical Transcriptions;” ..

Suppose we begin with the orthography of the written language.

e.g. CAT = [c] [a] [t] DOG = [d] [o] [g]

This implies• A relation between number of characters in a spelling and

the length of the pronunciation• The orthography of a language is consistent / efficient

Orthography Orthography asas PronunciationPronunciation

Implications for Data Implications for Data Labeling and TrainingLabeling and Training

Non-Roman Orthography: Non-Roman Orthography: Questions of TranscriptionQuestions of Transcription

Difficulties in Machine Difficulties in Machine Learning of PronunciationLearning of Pronunciation

“But there is a much more fundamental problem … in that it crucially assumes that letter-to-phoneme correspondences can in general be determined on the basis of information local to a particular portion of the letter string. While this is clearly true in some languages (e.g. Spanish), it is simply false for others….

“…It is unreasonable to expect that good results will be obtained from a system trained with no guidence of this kind, or … with data that is simply insufficient to the task.”

– Sproat et. al, Multilingual Text-to-Speech Synthesis: The Bell Labs Approach, pp.76-77

Lexicon / Letter-to-Sound Lexicon / Letter-to-Sound RulesRules

Token ProcessingToken Processing

Duration and Stress Duration and Stress ModelingModeling

Intonation and PhrasingIntonation and Phrasing

Unit Selection and Unit Selection and Waveform SynthesisWaveform Synthesis

Overview: Adaptation for Overview: Adaptation for Accent and DialectAccent and Dialect

Final PointsFinal Points