Return to Answer

If you can store pronunciations of all the phonemes in all the possible contexts and can smooth the transition between adjacent phonemes then you can copy-paste sounds. With a generative model we try model the human vocal tract system and depending on input context, speaking rate audio data is generated.

todo: Pick any word and record it in different context, different speaking rate, different speakers and plot waveforms. You'll see each time it has a different waveform. Save pronunciations of few phoneme and copy paste them to generate a word and listen to it.

If you can store pronunciations of all the phonemes in all the possible contexts and can smooth the transition between adjacent phonemes then you can copy-paste sounds. With a generative model we try model the human vocal tract system and depending on input context, speaking rate audio data is generated. Go through the first and third resource for detailed explanations.

todo: Pick any word and record it in different context, different speaking rate, different speakers and plot waveforms. You'll see each time it has a different waveform. Even if you don't change any condition each time there will be a slight variation in pronunciation of same word. Save pronunciations of few phoneme and copy paste them to generate a word and listen to it.

Edit 1: Some of the problems with "whenever you see this alphabet, pronounce it like this" are listed below

1. context: From the above to and go example which pronunciation of 'o' will be used

2. discontinuity: Vocal tract vibrations (lip, tongue motions) produce phoneme. Phonemes diffuse into neighboring phonemes since vocal tract doesn't stop vibrating immediately. If you copy-paste phonemes then there will be an abrupt change. You can copy-paste sounds for words since they are independent (kind of, 'can not' becomes 'can't' while speaking) and insert small pauses in between words. But then you have to store pronunciations for all the words.

3. stress phoneme: While speaking one or more phonemes are stressed (more focused, longer), Copy-pasting leaves out the stress information since everything is same

If you can store pronunciations of all the phonemes in all the possible contexts and can smooth the transition between adjacent phonemes then you can copy-paste sounds. With a generative model we try model the human vocal tract system and depending on input context, speaking rate audio data is generated.

All these problem arise because we want to make it sound more human. Phoneme transition and diffusion are the major hurdles.

todo: Pick any word and record it in different context, different speaking rate, different speakers and plot waveforms. You'll see each time it has a different waveform. Save pronunciations of few phoneme and copy paste them to generate a word and listen to it.

Speech data is made up of unique acoustic units called phonemes. Any audio file can be represented as a sequence of phonemes. Both automatic speech recognition (ASR) and speech synthesis (SS) systems model these phonemes. In ASR, speech signal (wav file) is used as input and phoneme labels are predicted and in SS, phoneme labels can be input and speech signal is output.

You can use a phonetic dictionary for converting your text files into sequence of phonemes. e.g play -> P L EY

If you have phoneme boundary marked data e.g. in audio file file1.wav 0.1s to 0.5s phoneme x and 0.5s to 0.9s is phoneme y. Now you have You can use a NN to learn the mapping between phoneme labels and speech signal (400 data points as output and phoneme label of these 400 points as input).

But there are many things that affect the pronunciation. Some of them are listed below:

Context: 'to' and 'go' have the same phoneme 'o' but have very different pronunciations.

Pitch: Female speakers usually have higher pitch than male speakers.

Speaking rate: Speaking rate varies across speakers. It also depends on speaking mode while reading a text we tend to have less number of pauses as compared to conversations.

length_of_output_phoneme: The length of wav file to generate

So in the end input to your NN will look something like this [left_context, phoneme, right_context, specking_rate, pitch, length_of_output_phoneme] and output will be corresponding speech signal. You can either use MFCC features or raw wav data as NN output. There are many other factors that affect the pronunciation.

If you don't have time marked data. You can use Hidden Markov model HMM for speech synthesis. A separate model will be learned for each phoneme. Input for HMM will be text files (sequence of phonemes) and output will be specch signal. These learned models can be used for generating speech data later.

Some speech synthesis resources are listed below:

1. CMU festvox

2. wavenet

3. Deep Learning in Speech Synthesis

The biggest challenge will be make it sound like human voice.

Speech data is made up of unique acoustic units called phonemes. Any audio file can be represented as a sequence of phonemes. Both automatic speech recognition (ASR) and speech synthesis (SS) systems model these phonemes. In ASR, speech signal (wav file) is used as input and phoneme labels are predicted and in SS, phoneme labels can be input and speech signal is output.

You can use a phonetic dictionary for converting your text files into sequence of phonemes. e.g play -> P L EY

If you have phoneme boundary marked data e.g. in audio file file1.wav 0.1s to 0.5s phoneme x and 0.5s to 0.9s is phoneme y. Now you have You can use a NN to learn the mapping between phoneme labels and speech signal (400 data points as output and phoneme label of these 400 points as input).

But there are many things that affect the pronunciation. Some of them are listed below:

Context: 'to' and 'go' have the same phoneme 'o' but have very different pronunciations.

Pitch: Female speakers usually have higher pitch than male speakers.

Speaking rate: Speaking rate varies across speakers. It also depends on speaking mode while reading a text we tend to have less number of pauses as compared to conversations.

length_of_output_phoneme: The length of wav file to generate

So in the end input to your NN will look something like this [left_context, phoneme, right_context, specking_rate, pitch, length_of_output_phoneme] and output will be corresponding speech signal. You can either use MFCC features or raw wav data as NN output. There are many other factors that affect the pronunciation.

If you don't have time marked data. You can use Hidden Markov model HMM for speech synthesis. A separate model will be learned for each phoneme. Input for HMM will be text files (sequence of phonemes) and output will be specch signal. These learned models can be used for generating speech data later.

Some speech synthesis resources are listed below:

1. CMU festvox

2. wavenet

3. Deep Learning in Speech Synthesis

The biggest challenge will be to make it sound like human voice.