python - What is the format of the WAV file for a Text to Speech Neural Network?

Question

I am creating a Text to Speech system for a phonetic language called "Kannada" and I plan to train it with a Neural Network. The input is a word/phrase while the output is the corresponding audio.

While implementing the Network, I was thinking the input should be the segmented characters of the word/phrase as the output pronunciation only depends on the characters that make up the word, unlike English where we have slient words and Part of Speech to consider. However, I do not know how I should train the output.

Since my Dataset is a collection of words/phrases and the corrusponding MP3 files, I thought of converting these files to WAV using pydub for all audio files.

from pydub import AudioSegment
sound = AudioSegment.from_mp3("audio/file1.mp3")
sound.export("wav/file1.wav", format="wav")

Next, I open the wav file and convert it to a normalized byte array with values between 0 and 1.

import numpy as np
import wave

f = wave.open('wav/kn3.wav', 'rb')
frames = f.readframes(-1)

#Array of integers of range [0,255]
data = np.fromstring(frames, dtype='uint8')

#Normalized bytes of wav
arr  = np.array(data)/255

How Should I train this?

From here, I am not sure how to train this with the input text. From this, I would need a variable number of input and output neurons in the First and Last layers as the number of characters (1st layer) and the bytes of the corresponding wave (Last layer) change for every input.

Since RNNs deal with such variable data, I thought it would come in handy here.

Correct me if I am wrong, but the output of Neural Networks are actually probability values between 0 and 1. However, we are not dealing with a classification problem. The audio can be anything, right? In my case, the "output" should be a vector of bytes corresponding to the WAV file. So there will be around 40,000 of these with values between 0 and 255 (without the normalization step) for every word. How do I train this speech data? Any suggestions are appreciated.

EDIT 1 : In response to arduinolover's Answer

From what I understand, Phonemes are the basic sounds of the language. So, why do I need a neural network to map phoneme labels with speech? Can't I just say, "whenever you see this alphabet, pronounce it like this". After all, this language, Kannada, is phonetic: There are no silent words. All words are pronounced the same way they are spelled. How would a Neural Network help here then?

On input of a new text, I just need to break it down to the corresponding alphabets (which are also the phonemes) and retrieve it's file (converted from WAV to raw byte data). Now, merge the bytes together and convert it to a wav file.

Is this this too simplistic? Am I missing something here? What would be the point of a Neural Network for this particular language (Kannada) ?

Answer 1

Speech data is made up of unique acoustic units called phonemes. Any audio file can be represented as a sequence of phonemes. Both automatic speech recognition (ASR) and speech synthesis (SS) systems model these phonemes. In ASR, speech signal (wav file) is used as input and phoneme labels are predicted and in SS, phoneme labels can be input and speech signal is output.

You can use a phonetic dictionary for converting your text files into sequence of phonemes. e.g play -> P L EY

If you have phoneme boundary marked data e.g. in audio file file1.wav 0.1s to 0.5s phoneme x and 0.5s to 0.9s is phoneme y. Now you have You can use a NN to learn the mapping between phoneme labels and speech signal (400 data points as output and phoneme label of these 400 points as input).

But there are many things that affect the pronunciation. Some of them are listed below:

Context: 'to' and 'go' have the same phoneme 'o' but have very different pronunciations.

Pitch: Female speakers usually have higher pitch than male speakers.

Speaking rate: Speaking rate varies across speakers. It also depends on speaking mode while reading a text we tend to have less number of pauses as compared to conversations.

length_of_output_phoneme: The length of wav file to generate

So in the end input to your NN will look something like this [left_context, phoneme, right_context, specking_rate, pitch, length_of_output_phoneme] and output will be corresponding speech signal. You can either use MFCC features or raw wav data as NN output. There are many other factors that affect the pronunciation.

If you don't have time marked data. You can use Hidden Markov model HMM for speech synthesis. A separate model will be learned for each phoneme. Input for HMM will be text files (sequence of phonemes) and output will be specch signal. These learned models can be used for generating speech data later.

Some speech synthesis resources are listed below:

1) CMU festvox

2) wavenet

3) Deep Learning in Speech Synthesis

The biggest challenge will be to make it sound like human voice.

Edit 1: Some of the problems with "whenever you see this alphabet, pronounce it like this" are listed below

1) context: From the above to and go example which pronunciation of 'o' will be used

2) discontinuity: Vocal tract vibrations (lip, tongue motions) produce phoneme. Phonemes diffuse into neighboring phonemes since vocal tract doesn't stop vibrating immediately. If you copy-paste phonemes then there will be an abrupt change. You can copy-paste sounds for words since they are independent (kind of, 'can not' becomes 'can't' while speaking) and insert small pauses in between words. But then you have to store pronunciations for all the words.

3) stress phoneme: While speaking one or more phonemes are stressed (more focused, longer), Copy-pasting leaves out the stress information since everything is same

If you can store pronunciations of all the phonemes in all the possible contexts and can smooth the transition between adjacent phonemes then you can copy-paste sounds. With a generative model we try model the human vocal tract system and depending on input context, speaking rate audio data is generated. Go through the first and third resource for detailed explanations.

All these problem arise because we want to make it sound more human. Phoneme transition and diffusion are the major hurdles.

todo: Pick any word and record it in different context, different speaking rate, different speakers and plot waveforms. You'll see each time it has a different waveform. Even if you don't change any condition each time there will be a slight variation in pronunciation of same word. Save pronunciations of few phoneme and copy paste them to generate a word and listen to it.