computer science student - asking for some machine learning guiding (voice cloning)

Question

I have chosen my synopsis topic for my second last semester.
I want to make a text-to-speech program, that speaks with the voice of a game character.

I have worked with machine learning in my class, but nothing that covers audio or text-to-speech.
I know how big of a project I have taken on my shoulders, but that is just what I need.

I don't know where to start at all, since there is way too much information out there. In other programming areas, I'm usually pretty good at finding information, but I just don't know enough about ML yet to do that here (plus the keywords used online kind of contradict each other).
I'm not sure which tools I need to look at. Where should I begin?

I have 68.871 audio files, which is about 76 hours of audio in total. The files are of different lengths, they start at around under a second and end around 49 mins. I have not heard all the audio myself, but I'm pretty sure all of it is completely clean.

(I know that I probably can't feed any model with 68.871 files, that could all be of different lengths, so I think I have to pick out a smaller sample size, with more equal lengths)

I would like to use these files as audio samples, so a model can learn how to speak like this character.

Any push in the right direction would be deeply appreciated.

Answer 1

Try to look into existing text-to-speech/speech synthesis. This link list multiple different models/approaches in this domain. When trying to look at state-of-the-art (SOTA) methods in this domain I find this GitHub page on which a lot of papers can be found on audio and speech. A very good (SOTA) model/architecture seems to be Tacotron 2/WaveNet. Hope this helps you with your project.